{"thread":{"id":"47524","subject":"[PATCH 00/26] protocol version 2","startedAt":"2018-01-03T00:18:48Z","lastAt":"2018-03-14T17:37:00Z","messageCount":362,"participants":["Brandon Williams","Stefan Beller","Jonathan Tan","Duy Nguyen","Derrick Stolee","Eric Sunshine","Jeff Hostetler","Randall S. Becker","'Brandon Williams'","Jonathan Nieder","Junio C Hamano","Jeff King","Ævar Arnfjörð Bjarmason","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":26},"messages":[{"id":"335646","messageId":"20180103001828.205012-1-bmwill@google.com","threadId":"47524","inReplyTo":null,"subject":"[PATCH 00/26] protocol version 2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:02Z","receivedAt":"2018-01-03T00:18:48Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The following patches extend what I sent out as an WIP\n(https://public-inbox.org/git/20171204235823.63299-1-bmwill@google.com/) and\nimplement protocol version 2.\n\nSome changes from that series are as follows:\n * Lots of various cleanup on the ls-refs and fetch command code, both server\n and client.\n * Fetch command now supports a stateless-rpc mode which enables communicating\n with a half-duplex connection.\n * Introduce a new remote-helper command 'connect-half-duplex' which is\n implemented by remote-curl (the http remote-helper). This allows for a\n client to establish a half-duplex connection and use remote-curl as a proxy\n to wrap requests in http before sending them to the remote end and\n unwrapping the responses and sending them back to the client's stdin.\n * The transport code is refactored for ls-remote, fetch, and push to provide a\n list of ref-patterns (based on the refspec being used) when requesting refs\n from the remote end. This allows the ls-refs code to send this list of\n patterns so the remote end and filter the refs it sends back.\n\nThis series effectively implements protocol version 2 for listing a remotes\nrefs (ls-remote) as well as for fetch for the builtin transports (ssh, git,\nfile) and for the http/https transports. Push is not implemented yet and\ndoesn't need to be implemented at the same time as fetch since the\nreceive-pack code can default to using protocol v0 when v2 is requested by the\nclient.\n\nAny feedback is appreciated! Thanks!\n\tBrandon\n\nBrandon Williams (26):\n pkt-line: introduce packet_read_with_status\n pkt-line: introduce struct packet_reader\n pkt-line: add delim packet support\n upload-pack: convert to a builtin\n upload-pack: factor out processing lines\n transport: use get_refs_via_connect to get refs\n connect: convert get_remote_heads to use struct packet_reader\n connect: discover protocol version outside of get_remote_heads\n transport: store protocol version\n protocol: introduce enum protocol_version value protocol_v2\n serve: introduce git-serve\n ls-refs: introduce ls-refs server command\n connect: request remote refs using v2\n transport: convert get_refs_list to take a list of ref patterns\n transport: convert transport_get_remote_refs to take a list of ref\n patterns\n ls-remote: pass ref patterns when requesting a remote's refs\n fetch: pass ref patterns when fetching\n push: pass ref patterns when pushing\n upload-pack: introduce fetch server command\n fetch-pack: perform a fetch using v2\n transport-helper: remove name parameter\n transport-helper: refactor process_connect_service\n transport-helper: introduce connect-half-duplex\n pkt-line: add packet_buf_write_len function\n remote-curl: create copy of the service name\n remote-curl: implement connect-half-duplex command\n\n .gitignore | 1 +\n Documentation/technical/protocol-v2.txt | 131 ++++++++++\n Makefile | 6 +-\n builtin.h | 2 +\n builtin/clone.c | 2 +-\n builtin/fetch-pack.c | 21 +-\n builtin/fetch.c | 14 +-\n builtin/ls-remote.c | 7 +-\n builtin/receive-pack.c | 6 +\n builtin/remote.c | 2 +-\n builtin/send-pack.c | 20 +-\n builtin/serve.c | 30 +++\n connect.c | 226 +++++++++++++-----\n connect.h | 3 +\n fetch-pack.c | 267 ++++++++++++++++++++-\n fetch-pack.h | 4 +-\n git.c | 2 +\n ls-refs.c | 97 ++++++++\n ls-refs.h | 9 +\n pkt-line.c | 147 +++++++++++-\n pkt-line.h | 76 ++++++\n protocol.c | 2 +\n protocol.h | 1 +\n remote-curl.c | 209 +++++++++++++++-\n remote.h | 9 +-\n serve.c | 243 +++++++++++++++++++\n serve.h | 15 ++\n t/t5701-protocol-v2.sh | 117 +++++++++\n transport-helper.c | 84 ++++---\n transport-internal.h | 4 +-\n transport.c | 119 ++++++---\n transport.h | 9 +-\n upload-pack.c | 412 ++++++++++++++++++++++++++++----\n upload-pack.h | 9 +\n 34 files changed, 2108 insertions(+), 198 deletions(-)\n create mode 100644 Documentation/technical/protocol-v2.txt\n create mode 100644 builtin/serve.c\n create mode 100644 ls-refs.c\n create mode 100644 ls-refs.h\n create mode 100644 serve.c\n create mode 100644 serve.h\n create mode 100755 t/t5701-protocol-v2.sh\n create mode 100644 upload-pack.h\n\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335647","messageId":"20180103001828.205012-2-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 01/26] pkt-line: introduce packet_read_with_status","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:03Z","receivedAt":"2018-01-03T00:18:50Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The current pkt-line API encodes the status of a pkt-line read in the\nlength of the read content. An error is indicated with '-1', a flush\nwith '0' (which can be confusing since a return value of '0' can also\nindicate an empty pkt-line), and a positive integer for the length of\nthe read content otherwise. This doesn't leave much room for allowing\nthe addition of additional special packets in the future.\n\nTo solve this introduce 'packet_read_with_status()' which reads a packet\nand returns the status of the read encoded as an 'enum packet_status'\ntype. This allows for easily identifying between special and normal\npackets as well as errors. It also enables easily adding a new special\npacket in the future.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 55 ++++++++++++++++++++++++++++++++++++++++++-------------\n pkt-line.h | 15 +++++++++++++++\n 2 files changed, 57 insertions(+), 13 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 2827ca772..8d7cd389f 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -280,28 +280,33 @@ static int packet_length(const char *linelen)\n \treturn (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);\n }\n \n-int packet_read(int fd, char **src_buf, size_t *src_len,\n-\t\tchar *buffer, unsigned size, int options)\n+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n+\t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n+\t\t\t\t\t\tint options)\n {\n-\tint len, ret;\n+\tint len;\n \tchar linelen[4];\n \n-\tret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);\n-\tif (ret < 0)\n-\t\treturn ret;\n+\tif (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)\n+\t\treturn PACKET_READ_EOF;\n+\n \tlen = packet_length(linelen);\n \tif (len < 0)\n \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n-\tif (!len) {\n+\n+\tif (len == 0) {\n \t\tpacket_trace(\"0000\", 4, 0);\n-\t\treturn 0;\n+\t\treturn PACKET_READ_FLUSH;\n+\t} else if (len >= 1 && len <= 3) {\n+\t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n \t}\n+\n \tlen -= 4;\n-\tif (len >= size)\n+\tif ((len < 0) || ((unsigned)len >= size))\n \t\tdie(\"protocol error: bad line length %d\", len);\n-\tret = get_packet_data(fd, src_buf, src_len, buffer, len, options);\n-\tif (ret < 0)\n-\t\treturn ret;\n+\n+\tif (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)\n+\t\treturn PACKET_READ_EOF;\n \n \tif ((options & PACKET_READ_CHOMP_NEWLINE) &&\n \t len && buffer[len-1] == '\\n')\n@@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,\n \n \tbuffer[len] = 0;\n \tpacket_trace(buffer, len, 0);\n-\treturn len;\n+\t*pktlen = len;\n+\treturn PACKET_READ_NORMAL;\n+}\n+\n+int packet_read(int fd, char **src_buffer, size_t *src_len,\n+\t\tchar *buffer, unsigned size, int options)\n+{\n+\tenum packet_read_status status;\n+\tint pktlen;\n+\n+\tstatus = packet_read_with_status(fd, src_buffer, src_len,\n+\t\t\t\t\t buffer, size, &pktlen,\n+\t\t\t\t\t options);\n+\tswitch (status) {\n+\tcase PACKET_READ_EOF:\n+\t\tpktlen = -1;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\tbreak;\n+\tcase PACKET_READ_FLUSH:\n+\t\tpktlen = 0;\n+\t\tbreak;\n+\t}\n+\n+\treturn pktlen;\n }\n \n static char *packet_read_line_generic(int fd,\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3dad583e2..06c468927 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n int packet_read(int fd, char **src_buffer, size_t *src_len, char\n \t\t*buffer, unsigned size, int options);\n \n+/*\n+ * Read a packetized line into a buffer like the 'packet_read()' function but\n+ * returns an 'enum packet_read_status' which indicates the status of the read.\n+ * The number of bytes read will be assigined to *pktlen if the status of the\n+ * read was 'PACKET_READ_NORMAL'.\n+ */\n+enum packet_read_status {\n+\tPACKET_READ_EOF = -1,\n+\tPACKET_READ_NORMAL,\n+\tPACKET_READ_FLUSH,\n+};\n+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n+\t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n+\t\t\t\t\t\tint options);\n+\n /*\n * Convenience wrapper for packet_read that is not gentle, and sets the\n * CHOMP_NEWLINE option. The return value is NULL for a flush packet,\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335648","messageId":"20180103001828.205012-3-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 02/26] pkt-line: introduce struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:04Z","receivedAt":"2018-01-03T00:18:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Sometimes it is advantageous to be able to peek the next packet line\nwithout consuming it (e.g. to be able to determine the protocol version\na server is speaking). In order to do that introduce 'struct\npacket_reader' which is an abstraction around the normal packet reading\nlogic. This enables a caller to be able to peek a single line at a time\nusing 'packet_reader_peek()' and having a caller consume a line by\ncalling 'packet_reader_read()'.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n pkt-line.h | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 116 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 8d7cd389f..98c2d7d68 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -406,3 +406,62 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)\n \t}\n \treturn sb_out->len - orig_len;\n }\n+\n+/* Packet Reader Functions */\n+void packet_reader_init(struct packet_reader *reader, int fd,\n+\t\t\tchar *src_buffer, size_t src_len,\n+\t\t\tint options)\n+{\n+\tmemset(reader, 0, sizeof(*reader));\n+\n+\treader->fd = fd;\n+\treader->src_buffer = src_buffer;\n+\treader->src_len = src_len;\n+\treader->buffer = packet_buffer;\n+\treader->buffer_size = sizeof(packet_buffer);\n+\treader->options = options;\n+}\n+\n+enum packet_read_status packet_reader_read(struct packet_reader *reader)\n+{\n+\tif (reader->line_peeked) {\n+\t\treader->line_peeked = 0;\n+\t\treturn reader->status;\n+\t}\n+\n+\treader->status = packet_read_with_status(reader->fd,\n+\t\t\t\t\t\t &reader->src_buffer,\n+\t\t\t\t\t\t &reader->src_len,\n+\t\t\t\t\t\t reader->buffer,\n+\t\t\t\t\t\t reader->buffer_size,\n+\t\t\t\t\t\t &reader->pktlen,\n+\t\t\t\t\t\t reader->options);\n+\n+\tswitch (reader->status) {\n+\tcase PACKET_READ_EOF:\n+\t\treader->pktlen = -1;\n+\t\treader->line = NULL;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\treader->line = reader->buffer;\n+\t\tbreak;\n+\tcase PACKET_READ_FLUSH:\n+\t\treader->pktlen = 0;\n+\t\treader->line = NULL;\n+\t\tbreak;\n+\t}\n+\n+\treturn reader->status;\n+}\n+\n+enum packet_read_status packet_reader_peek(struct packet_reader *reader)\n+{\n+\t/* Only allow peeking a single line */\n+\tif (reader->line_peeked)\n+\t\treturn reader->status;\n+\n+\t/* Peek a line by reading it and setting peeked flag */\n+\tpacket_reader_read(reader);\n+\treader->line_peeked = 1;\n+\treturn reader->status;\n+}\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 06c468927..c446e886a 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -111,6 +111,63 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n */\n ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);\n \n+struct packet_reader {\n+\t/* source file descriptor */\n+\tint fd;\n+\n+\t/* source buffer and its size */\n+\tchar *src_buffer;\n+\tsize_t src_len;\n+\n+\t/* buffer that pkt-lines are read into and its size */\n+\tchar *buffer;\n+\tunsigned buffer_size;\n+\n+\t/* options to be used during reads */\n+\tint options;\n+\n+\t/* status of the last read */\n+\tenum packet_read_status status;\n+\n+\t/* length of data read during the last read */\n+\tint pktlen;\n+\n+\t/* the last line read */\n+\tconst char *line;\n+\n+\t/* indicates if a line has been peeked */\n+\tint line_peeked;\n+};\n+\n+/*\n+ * Initialize a 'struct packet_reader' object which is an\n+ * abstraction around the 'packet_read_with_status()' function.\n+ */\n+extern void packet_reader_init(struct packet_reader *reader, int fd,\n+\t\t\t char *src_buffer, size_t src_len,\n+\t\t\t int options);\n+\n+/*\n+ * Perform a packet read and return the status of the read.\n+ * The values of 'pktlen' and 'line' are updated based on the status of the\n+ * read as follows:\n+ *\n+ * PACKET_READ_ERROR: 'pktlen' is set to '-1' and 'line' is set to NULL\n+ * PACKET_READ_NORMAL: 'pktlen' is set to the number of bytes read\n+ *\t\t 'line' is set to point at the read line\n+ * PACKET_READ_FLUSH: 'pktlen' is set to '0' and 'line' is set to NULL\n+ */\n+extern enum packet_read_status packet_reader_read(struct packet_reader *reader);\n+\n+/*\n+ * Peek the next packet line without consuming it and return the status.\n+ * The next call to 'packet_reader_read()' will perform a read of the same line\n+ * that was peeked, consuming the line.\n+ *\n+ * Only a single line can be peeked at a time.\n+ */\n+extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);\n+\n #define DEFAULT_PACKET_MAX 1000\n #define LARGE_PACKET_MAX 65520\n #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335649","messageId":"20180103001828.205012-6-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 05/26] upload-pack: factor out processing lines","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:07Z","receivedAt":"2018-01-03T00:18:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Factor out the logic for processing shallow, deepen, deepen_since, and\ndeepen_not lines into their own functions to simplify the\n'receive_needs()' function in addition to making it easier to reuse some\nof this logic when implementing protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 74 insertions(+), 39 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 20acaa49d..9a507ae53 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -731,6 +731,75 @@ static void deepen_by_rev_list(int ac, const char **av,\n \tpacket_flush(1);\n }\n \n+static int process_shallow(const char *line, struct object_array *shallows)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"shallow \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tstruct object *object;\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"invalid shallow line: %s\", line);\n+\t\tobject = parse_object(&oid);\n+\t\tif (!object)\n+\t\t\treturn 1;\n+\t\tif (object->type != OBJ_COMMIT)\n+\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n+\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n+\t\t\tobject->flags |= CLIENT_SHALLOW;\n+\t\t\tadd_object_array(object, NULL, shallows);\n+\t\t}\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_deepen(const char *line, int *depth)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\t*depth = strtol(arg, &end, 0);\n+\t\tif (!end || *end || depth <= 0)\n+\t\t\tdie(\"Invalid deepen: %s\", line);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen-since \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\t*deepen_since = parse_timestamp(arg, &end, 0);\n+\t\tif (!end || *end || !deepen_since ||\n+\t\t /* revisions.c's max_age -1 is special */\n+\t\t *deepen_since == -1)\n+\t\t\tdie(\"Invalid deepen-since: %s\", line);\n+\t\t*deepen_rev_list = 1;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen-not \", &arg)) {\n+\t\tchar *ref = NULL;\n+\t\tstruct object_id oid;\n+\t\tif (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n+\t\t\tdie(\"git upload-pack: ambiguous deepen-not: %s\", line);\n+\t\tstring_list_append(deepen_not, ref);\n+\t\tfree(ref);\n+\t\t*deepen_rev_list = 1;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static void receive_needs(void)\n {\n \tstruct object_array shallows = OBJECT_ARRAY_INIT;\n@@ -752,49 +821,15 @@ static void receive_needs(void)\n \t\tif (!line)\n \t\t\tbreak;\n \n-\t\tif (skip_prefix(line, \"shallow \", &arg)) {\n-\t\t\tstruct object_id oid;\n-\t\t\tstruct object *object;\n-\t\t\tif (get_oid_hex(arg, &oid))\n-\t\t\t\tdie(\"invalid shallow line: %s\", line);\n-\t\t\tobject = parse_object(&oid);\n-\t\t\tif (!object)\n-\t\t\t\tcontinue;\n-\t\t\tif (object->type != OBJ_COMMIT)\n-\t\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n-\t\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n-\t\t\t\tobject->flags |= CLIENT_SHALLOW;\n-\t\t\t\tadd_object_array(object, NULL, &shallows);\n-\t\t\t}\n+\t\tif (process_shallow(line, &shallows))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen \", &arg)) {\n-\t\t\tchar *end = NULL;\n-\t\t\tdepth = strtol(arg, &end, 0);\n-\t\t\tif (!end || *end || depth <= 0)\n-\t\t\t\tdie(\"Invalid deepen: %s\", line);\n+\t\tif (process_deepen(line, &depth))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen-since \", &arg)) {\n-\t\t\tchar *end = NULL;\n-\t\t\tdeepen_since = parse_timestamp(arg, &end, 0);\n-\t\t\tif (!end || *end || !deepen_since ||\n-\t\t\t /* revisions.c's max_age -1 is special */\n-\t\t\t deepen_since == -1)\n-\t\t\t\tdie(\"Invalid deepen-since: %s\", line);\n-\t\t\tdeepen_rev_list = 1;\n+\t\tif (process_deepen_since(line, &deepen_since, &deepen_rev_list))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen-not \", &arg)) {\n-\t\t\tchar *ref = NULL;\n-\t\t\tstruct object_id oid;\n-\t\t\tif (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n-\t\t\t\tdie(\"git upload-pack: ambiguous deepen-not: %s\", line);\n-\t\t\tstring_list_append(&deepen_not, ref);\n-\t\t\tfree(ref);\n-\t\t\tdeepen_rev_list = 1;\n+\t\tif (process_deepen_not(line, &deepen_not, &deepen_rev_list))\n \t\t\tcontinue;\n-\t\t}\n+\n \t\tif (!skip_prefix(line, \"want \", &arg) ||\n \t\t get_oid_hex(arg, &oid_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335650","messageId":"20180103001828.205012-8-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:09Z","receivedAt":"2018-01-03T00:18:57Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to allow for better control flow when protocol_v2 is introduced\nconvert 'get_remote_heads()' to use 'struct packet_reader' to read\npacket lines. This enables a client to be able to peek the first line\nof a server's response (without consuming it) in order to determine the\nprotocol version its speaking and then passing control to the\nappropriate handler.\n\nThis is needed because the initial response from a server speaking\nprotocol_v0 includes the first ref, while subsequent protocol versions\nrespond with a version line. We want to be able to read this first line\nwithout consuming the first ref sent in the protocol_v0 case so that the\nprotocol version the server is speaking can be determined outside of\n'get_remote_heads()' in a future patch.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 127 +++++++++++++++++++++++++++++++++++---------------------------\n 1 file changed, 72 insertions(+), 55 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex c3a014c5b..03bbb74e4 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -48,6 +48,12 @@ int check_ref_type(const struct ref *ref, int flags)\n \n static void die_initial_contact(int unexpected)\n {\n+\t/*\n+\t * A hang-up after seeing some response from the other end\n+\t * means that it is unexpected, as we know the other end is\n+\t * willing to talk to us. A hang-up before seeing any\n+\t * response does not necessarily mean an ACL problem, though.\n+\t */\n \tif (unexpected)\n \t\tdie(_(\"The remote end hung up upon initial contact\"));\n \telse\n@@ -56,6 +62,41 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n+static enum protocol_version discover_version(struct packet_reader *reader)\n+{\n+\tenum protocol_version version = protocol_unknown_version;\n+\n+\t/*\n+\t * Peek the first line of the server's response to\n+\t * determine the protocol version the server is speaking.\n+\t */\n+\tswitch (packet_reader_peek(reader)) {\n+\tcase PACKET_READ_EOF:\n+\t\tdie_initial_contact(0);\n+\tcase PACKET_READ_FLUSH:\n+\tcase PACKET_READ_DELIM:\n+\t\tversion = protocol_v0;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\tversion = determine_protocol_version_client(reader->line);\n+\t\tbreak;\n+\t}\n+\n+\t/* Maybe process capabilities here, at least for v2 */\n+\tswitch (version) {\n+\tcase protocol_v1:\n+\t\t/* Read the peeked version line */\n+\t\tpacket_reader_read(reader);\n+\t\tbreak;\n+\tcase protocol_v0:\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tdie(\"unknown protocol version: '%s'\\n\", reader->line);\n+\t}\n+\n+\treturn version;\n+}\n+\n static void parse_one_symref_info(struct string_list *symref, const char *val, int len)\n {\n \tchar *sym, *target;\n@@ -109,44 +150,10 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n-/*\n- * Read one line of a server's ref advertisement into packet_buffer.\n- */\n-static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n-\t\t\t int *responded)\n-{\n-\tint len = packet_read(in, src_buf, src_len,\n-\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\tconst char *arg;\n-\tif (len < 0)\n-\t\tdie_initial_contact(*responded);\n-\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n-\t\tdie(\"remote error: %s\", arg);\n-\n-\t*responded = 1;\n-\n-\treturn len;\n-}\n-\n-#define EXPECTING_PROTOCOL_VERSION 0\n-#define EXPECTING_FIRST_REF 1\n-#define EXPECTING_REF 2\n-#define EXPECTING_SHALLOW 3\n-\n-/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n-static int process_protocol_version(void)\n-{\n-\tswitch (determine_protocol_version_client(packet_buffer)) {\n-\tcase protocol_v1:\n-\t\treturn 1;\n-\tcase protocol_v0:\n-\t\treturn 0;\n-\tdefault:\n-\t\tdie(\"server is speaking an unknown protocol\");\n-\t}\n-}\n+#define EXPECTING_FIRST_REF 0\n+#define EXPECTING_REF 1\n+#define EXPECTING_SHALLOW 2\n+#define EXPECTING_DONE 3\n \n static void process_capabilities(int *len)\n {\n@@ -230,28 +237,36 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t struct oid_array *shallow_points)\n {\n \tstruct ref **orig_list = list;\n+\tint len = 0;\n+\tint state = EXPECTING_FIRST_REF;\n+\tstruct packet_reader reader;\n+\tconst char *arg;\n \n-\t/*\n-\t * A hang-up after seeing some response from the other end\n-\t * means that it is unexpected, as we know the other end is\n-\t * willing to talk to us. A hang-up before seeing any\n-\t * response does not necessarily mean an ACL problem, though.\n-\t */\n-\tint responded = 0;\n-\tint len;\n-\tint state = EXPECTING_PROTOCOL_VERSION;\n+\tpacket_reader_init(&reader, in, src_buf, src_len,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tdiscover_version(&reader);\n \n \t*list = NULL;\n \n-\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\twhile (state != EXPECTING_DONE) {\n+\t\tswitch (packet_reader_read(&reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tdie_initial_contact(1);\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tlen = reader.pktlen;\n+\t\t\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n+\t\t\t\tdie(\"remote error: %s\", arg);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tstate = EXPECTING_DONE;\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tdie(\"invalid packet\\n\");\n+\t\t}\n+\n \t\tswitch (state) {\n-\t\tcase EXPECTING_PROTOCOL_VERSION:\n-\t\t\tif (process_protocol_version()) {\n-\t\t\t\tstate = EXPECTING_FIRST_REF;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tstate = EXPECTING_FIRST_REF;\n-\t\t\t/* fallthrough */\n \t\tcase EXPECTING_FIRST_REF:\n \t\t\tprocess_capabilities(&len);\n \t\t\tif (process_dummy_ref()) {\n@@ -269,6 +284,8 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\tif (process_shallow(len, shallow_points))\n \t\t\t\tbreak;\n \t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n+\t\tcase EXPECTING_DONE:\n+\t\t\tbreak;\n \t\tdefault:\n \t\t\tdie(\"unexpected state %d\", state);\n \t\t}\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335651","messageId":"20180103001828.205012-9-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 08/26] connect: discover protocol version outside of get_remote_heads","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:10Z","receivedAt":"2018-01-03T00:19:03Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to prepare for the addition of protocol_v2 push the protocol\nversion discovery outside of 'get_remote_heads()'. This will allow for\nkeeping the logic for processing the reference advertisement for\nprotocol_v1 and protocol_v0 separate from the logic for protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 16 +++++++++++++++-\n builtin/send-pack.c | 17 +++++++++++++++--\n connect.c | 15 ++++-----------\n connect.h | 3 +++\n remote-curl.c | 20 ++++++++++++++++++--\n remote.h | 5 +++--\n transport.c | 24 +++++++++++++++++++-----\n 7 files changed, 77 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 366b9d13f..85d4faf76 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -4,6 +4,7 @@\n #include \"remote.h\"\n #include \"connect.h\"\n #include \"sha1-array.h\"\n+#include \"protocol.h\"\n \n static const char fetch_pack_usage[] =\n \"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] \"\n@@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tstruct fetch_pack_args args;\n \tstruct oid_array shallow = OID_ARRAY_INIT;\n \tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n+\tstruct packet_reader reader;\n \n \tpacket_trace_identity(\"fetch-pack\");\n \n@@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\tif (!conn)\n \t\t\treturn args.diag_url ? 0 : 1;\n \t}\n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n+\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n \t\t\t &shallow, pack_lockfile_ptr);\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex fc4f0bb5f..83cb125a6 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -14,6 +14,7 @@\n #include \"sha1-array.h\"\n #include \"gpg-interface.h\"\n #include \"gettext.h\"\n+#include \"protocol.h\"\n \n static const char * const send_pack_usage[] = {\n \tN_(\"git send-pack [--all | --mirror] [--dry-run] [--force] \"\n@@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tstruct packet_reader reader;\n \n \tstruct option options[] = {\n \t\tOPT__VERBOSITY(&verbose),\n@@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n-\t\t\t &extra_have, &shallow);\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n+\t\t\t\t &extra_have, &shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/connect.c b/connect.c\nindex 03bbb74e4..1787b0212 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n-static enum protocol_version discover_version(struct packet_reader *reader)\n+enum protocol_version discover_version(struct packet_reader *reader)\n {\n \tenum protocol_version version = protocol_unknown_version;\n \n@@ -231,7 +231,7 @@ static int process_shallow(int len, struct oid_array *shallow_points)\n /*\n * Read all the refs from the other end\n */\n-struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n+struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t struct ref **list, unsigned int flags,\n \t\t\t struct oid_array *extra_have,\n \t\t\t struct oid_array *shallow_points)\n@@ -239,23 +239,16 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \tstruct ref **orig_list = list;\n \tint len = 0;\n \tint state = EXPECTING_FIRST_REF;\n-\tstruct packet_reader reader;\n \tconst char *arg;\n \n-\tpacket_reader_init(&reader, in, src_buf, src_len,\n-\t\t\t PACKET_READ_CHOMP_NEWLINE |\n-\t\t\t PACKET_READ_GENTLE_ON_EOF);\n-\n-\tdiscover_version(&reader);\n-\n \t*list = NULL;\n \n \twhile (state != EXPECTING_DONE) {\n-\t\tswitch (packet_reader_read(&reader)) {\n+\t\tswitch (packet_reader_read(reader)) {\n \t\tcase PACKET_READ_EOF:\n \t\t\tdie_initial_contact(1);\n \t\tcase PACKET_READ_NORMAL:\n-\t\t\tlen = reader.pktlen;\n+\t\t\tlen = reader->pktlen;\n \t\t\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n \t\t\t\tdie(\"remote error: %s\", arg);\n \t\t\tbreak;\ndiff --git a/connect.h b/connect.h\nindex 01f14cdf3..cdb8979dc 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);\n extern const char *server_feature_value(const char *feature, int *len_ret);\n extern int url_is_local_not_ssh(const char *url);\n \n+struct packet_reader;\n+extern enum protocol_version discover_version(struct packet_reader *reader);\n+\n #endif\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 0053b0954..9f6d07683 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"remote.h\"\n+#include \"connect.h\"\n #include \"strbuf.h\"\n #include \"walker.h\"\n #include \"http.h\"\n@@ -13,6 +14,7 @@\n #include \"credential.h\"\n #include \"sha1-array.h\"\n #include \"send-pack.h\"\n+#include \"protocol.h\"\n \n static struct remote *remote;\n /* always ends with a trailing slash */\n@@ -176,8 +178,22 @@ static struct discovery *last_discovery;\n static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n-\tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n+\tstruct packet_reader reader;\n+\n+\tpacket_reader_init(&reader, -1, heads->buf, heads->len,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n+\t\t\t\t NULL, &heads->shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \treturn list;\n }\n \ndiff --git a/remote.h b/remote.h\nindex 1f6611be2..2016461df 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n void free_refs(struct ref *ref);\n \n struct oid_array;\n-extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n+struct packet_reader;\n+extern struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t\t struct ref **list, unsigned int flags,\n \t\t\t\t struct oid_array *extra_have,\n-\t\t\t\t struct oid_array *shallow);\n+\t\t\t\t struct oid_array *shallow_points);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\ndiff --git a/transport.c b/transport.c\nindex 8e8779096..63c3dbab9 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -18,6 +18,7 @@\n #include \"sha1-array.h\"\n #include \"sigchain.h\"\n #include \"transport-internal.h\"\n+#include \"protocol.h\"\n \n static void set_upstreams(struct transport *transport, struct ref *refs,\n \tint pretend)\n@@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)\n static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n {\n \tstruct git_transport_data *data = transport->data;\n-\tstruct ref *refs;\n+\tstruct ref *refs = NULL;\n+\tstruct packet_reader reader;\n \n \tconnect_setup(transport, for_push);\n-\tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0,\n-\t\t\t &data->extra_have,\n-\t\t\t &data->shallow);\n+\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &refs,\n+\t\t\t\t for_push ? REF_NORMAL : 0,\n+\t\t\t\t &data->extra_have,\n+\t\t\t\t &data->shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335652","messageId":"20180103001828.205012-4-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 03/26] pkt-line: add delim packet support","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:05Z","receivedAt":"2018-01-03T00:19:05Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"One of the design goals of protocol-v2 is to improve the semantics of\nflush packets. Currently in protocol-v1, flush packets are used both to\nindicate a break in a list of packet lines as well as an indication that\none side has finished speaking. This makes it particularly difficult\nto implement proxies as a proxy would need to completely understand git\nprotocol instead of simply looking for a flush packet.\n\nTo do this, introduce the special deliminator packet '0001'. A delim\npacket can then be used as a deliminator between lists of packet lines\nwhile flush packets can be reserved to indicate the end of a response.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 19 ++++++++++++++++++-\n pkt-line.h | 3 +++\n 2 files changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 98c2d7d68..3159cbe10 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -91,6 +91,12 @@ void packet_flush(int fd)\n \twrite_or_die(fd, \"0000\", 4);\n }\n \n+void packet_delim(int fd)\n+{\n+\tpacket_trace(\"0001\", 4, 1);\n+\twrite_or_die(fd, \"0001\", 4);\n+}\n+\n int packet_flush_gently(int fd)\n {\n \tpacket_trace(\"0000\", 4, 1);\n@@ -105,6 +111,12 @@ void packet_buf_flush(struct strbuf *buf)\n \tstrbuf_add(buf, \"0000\", 4);\n }\n \n+void packet_buf_delim(struct strbuf *buf)\n+{\n+\tpacket_trace(\"0001\", 4, 1);\n+\tstrbuf_add(buf, \"0001\", 4);\n+}\n+\n static void set_packet_header(char *buf, const int size)\n {\n \tstatic char hexchar[] = \"0123456789abcdef\";\n@@ -297,7 +309,10 @@ enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_\n \tif (len == 0) {\n \t\tpacket_trace(\"0000\", 4, 0);\n \t\treturn PACKET_READ_FLUSH;\n-\t} else if (len >= 1 && len <= 3) {\n+\t} else if (len == 1) {\n+\t\tpacket_trace(\"0001\", 4, 0);\n+\t\treturn PACKET_READ_DELIM;\n+\t} else if (len >= 2 && len <= 3) {\n \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n \t}\n \n@@ -333,6 +348,7 @@ int packet_read(int fd, char **src_buffer, size_t *src_len,\n \t\tbreak;\n \tcase PACKET_READ_NORMAL:\n \t\tbreak;\n+\tcase PACKET_READ_DELIM:\n \tcase PACKET_READ_FLUSH:\n \t\tpktlen = 0;\n \t\tbreak;\n@@ -445,6 +461,7 @@ enum packet_read_status packet_reader_read(struct packet_reader *reader)\n \tcase PACKET_READ_NORMAL:\n \t\treader->line = reader->buffer;\n \t\tbreak;\n+\tcase PACKET_READ_DELIM:\n \tcase PACKET_READ_FLUSH:\n \t\treader->pktlen = 0;\n \t\treader->line = NULL;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex c446e886a..97b6dd1c7 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -20,8 +20,10 @@\n * side can't, we stay with pure read/write interfaces.\n */\n void packet_flush(int fd);\n+void packet_delim(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n+void packet_buf_delim(struct strbuf *buf);\n void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n@@ -75,6 +77,7 @@ enum packet_read_status {\n \tPACKET_READ_EOF = -1,\n \tPACKET_READ_NORMAL,\n \tPACKET_READ_FLUSH,\n+\tPACKET_READ_DELIM,\n };\n enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n \t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335653","messageId":"20180103001828.205012-14-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 13/26] connect: request remote refs using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:15Z","receivedAt":"2018-01-03T00:19:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach the client to be able to request a remote's refs using protocol\nv2. This is done by having a client issue a 'ls-refs' request to a v2\nserver.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++-\n remote.h | 4 ++\n t/t5701-protocol-v2.sh | 28 ++++++++++++++\n transport.c | 2 +-\n upload-pack.c | 9 +++--\n 5 files changed, 138 insertions(+), 6 deletions(-)\n create mode 100755 t/t5701-protocol-v2.sh\n\ndiff --git a/connect.c b/connect.c\nindex caa539b75..9badd403f 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -12,9 +12,11 @@\n #include \"sha1-array.h\"\n #include \"transport.h\"\n #include \"strbuf.h\"\n+#include \"version.h\"\n #include \"protocol.h\"\n \n static char *server_capabilities;\n+static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n static const char *parse_feature_value(const char *, const char *, int *);\n \n static int check_ref(const char *name, unsigned int flags)\n@@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n+static int server_supports_v2(const char *c, int die_on_error)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n+\t\t (!*out || *out == '='))\n+\t\t\treturn 1;\n+\t}\n+\n+\tif (die_on_error)\n+\t\tdie(\"server doesn't support '%s'\", c);\n+\n+\treturn 0;\n+}\n+\n+static void process_capabilities_v2(struct packet_reader *reader)\n+{\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\targv_array_push(&server_capabilities_v2, reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\tdie(\"protocol error\");\n+}\n+\n enum protocol_version discover_version(struct packet_reader *reader)\n {\n \tenum protocol_version version = protocol_unknown_version;\n@@ -85,7 +114,7 @@ enum protocol_version discover_version(struct packet_reader *reader)\n \t/* Maybe process capabilities here, at least for v2 */\n \tswitch (version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tprocess_capabilities_v2(reader);\n \t\tbreak;\n \tcase protocol_v1:\n \t\t/* Read the peeked version line */\n@@ -292,6 +321,76 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n \treturn list;\n }\n \n+static int process_ref_v2(const char *line, struct ref ***list)\n+{\n+\tint ret = 1;\n+\tint i = 0;\n+\tstruct object_id old_oid;\n+\tstruct ref *ref;\n+\tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n+\n+\tif (string_list_split(&line_sections, line, ' ', -1) < 2) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tif (get_oid_hex(line_sections.items[i++].string, &old_oid)) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tref = alloc_ref(line_sections.items[i++].string);\n+\n+\tif (i < line_sections.nr)\n+\t\tref->symref = xstrdup(line_sections.items[i++].string);\n+\n+\toidcpy(&ref->old_oid, &old_oid);\n+\t**list = ref;\n+\t*list = &ref->next;\n+\n+out:\n+\tstring_list_clear(&line_sections, 0);\n+\treturn ret;\n+}\n+\n+struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n+\t\t\t struct ref **list, int for_push,\n+\t\t\t const struct argv_array *ref_patterns)\n+{\n+\tint i;\n+\t*list = NULL;\n+\n+\t/* Check that the server supports the ls-refs command */\n+\t/* Issue request for ls-refs */\n+\tif (server_supports_v2(\"ls-refs\", 1))\n+\t\tpacket_write_fmt(fd_out, \"command=ls-refs\\n\");\n+\n+\tif (server_supports_v2(\"agent\", 0))\n+\t packet_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n+\n+\tpacket_delim(fd_out);\n+\t/* When pushing we don't want to request the peeled tags */\n+\tif (!for_push)\n+\t\tpacket_write_fmt(fd_out, \"peel\\n\");\n+\tpacket_write_fmt(fd_out, \"symrefs\\n\");\n+\tfor (i = 0; ref_patterns && i < ref_patterns->argc; i++) {\n+\t\tpacket_write_fmt(fd_out, \"ref-pattern %s\\n\",\n+\t\t\t\t ref_patterns->argv[i]);\n+\t}\n+\tpacket_flush(fd_out);\n+\n+\t/* Process response from server */\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tif (!process_ref_v2(reader->line, &list))\n+\t\t\tdie(\"invalid ls-refs response: %s\", reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\tdie(\"protocol error\");\n+\n+\treturn list;\n+}\n+\n static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)\n {\n \tint len;\ndiff --git a/remote.h b/remote.h\nindex 2016461df..21d0c776c 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -151,10 +151,14 @@ void free_refs(struct ref *ref);\n \n struct oid_array;\n struct packet_reader;\n+struct argv_array;\n extern struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t\t struct ref **list, unsigned int flags,\n \t\t\t\t struct oid_array *extra_have,\n \t\t\t\t struct oid_array *shallow_points);\n+extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n+\t\t\t\t struct ref **list, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\ndiff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh\nnew file mode 100755\nindex 000000000..4bf4d61ac\n--- /dev/null\n+++ b/t/t5701-protocol-v2.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='test git wire-protocol version 2'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'file://' transport\n+#\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init file_parent &&\n+\ttest_commit -C file_parent one\n+'\n+\n+test_expect_success 'list refs with file:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tls-remote --symref \"file://$(pwd)/file_parent\" >actual 2>log &&\n+\n+\t# Server responded using protocol v2\n+\tcat log &&\n+\tgrep \"git< version 2\" log &&\n+\n+\tgit ls-remote --symref \"file://$(pwd)/file_parent\" >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/transport.c b/transport.c\nindex 83d9dd1df..ffc6b2614 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -204,7 +204,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 2bc888fc1..2ca60d27c 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -19,6 +19,7 @@\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n #include \"protocol.h\"\n+#include \"serve.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -1071,6 +1072,7 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n {\n \tconst char *dir;\n \tint strict = 0;\n+\tstruct serve_options opts = SERVE_OPTIONS_INIT;\n \tstruct option options[] = {\n \t\tOPT_BOOL(0, \"stateless-rpc\", &stateless_rpc,\n \t\t\t N_(\"quit after a single request/response exchange\")),\n@@ -1105,10 +1107,9 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \n \tswitch (determine_protocol_version_server()) {\n \tcase protocol_v2:\n-\t\t/*\n-\t\t * fetch support for protocol v2 has not been implemented yet,\n-\t\t * so ignore the request to use v2 and fallback to using v0.\n-\t\t */\n+\t\topts.advertise_capabilities = advertise_refs;\n+\t\topts.stateless_rpc = stateless_rpc;\n+\t\tserve(&opts);\n \t\tbreak;\n \tcase protocol_v1:\n \t\t/*\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335654","messageId":"20180103001828.205012-5-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 04/26] upload-pack: convert to a builtin","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:06Z","receivedAt":"2018-01-03T00:19:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to allow for code sharing with the server-side of fetch in\nprotocol-v2 convert upload-pack to be a builtin.\n\nSigned-off-by: Brandon Williams \n---\n Makefile | 3 ++-\n builtin.h | 1 +\n git.c | 1 +\n upload-pack.c | 2 +-\n 4 files changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 2a81ae22e..e0740b452 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -636,7 +636,6 @@ PROGRAM_OBJS += imap-send.o\n PROGRAM_OBJS += sh-i18n--envsubst.o\n PROGRAM_OBJS += shell.o\n PROGRAM_OBJS += show-index.o\n-PROGRAM_OBJS += upload-pack.o\n PROGRAM_OBJS += remote-testsvn.o\n \n # Binary suffix, set to .exe for Windows builds\n@@ -701,6 +700,7 @@ BUILT_INS += git-merge-subtree$X\n BUILT_INS += git-show$X\n BUILT_INS += git-stage$X\n BUILT_INS += git-status$X\n+BUILT_INS += git-upload-pack$X\n BUILT_INS += git-whatchanged$X\n \n # what 'all' will build and 'install' will install in gitexecdir,\n@@ -904,6 +904,7 @@ LIB_OBJS += tree-diff.o\n LIB_OBJS += tree.o\n LIB_OBJS += tree-walk.o\n LIB_OBJS += unpack-trees.o\n+LIB_OBJS += upload-pack.o\n LIB_OBJS += url.o\n LIB_OBJS += urlmatch.o\n LIB_OBJS += usage.o\ndiff --git a/builtin.h b/builtin.h\nindex 42378f3aa..f332a1257 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -231,6 +231,7 @@ extern int cmd_update_ref(int argc, const char **argv, const char *prefix);\n extern int cmd_update_server_info(int argc, const char **argv, const char *prefix);\n extern int cmd_upload_archive(int argc, const char **argv, const char *prefix);\n extern int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix);\n+extern int cmd_upload_pack(int argc, const char **argv, const char *prefix);\n extern int cmd_var(int argc, const char **argv, const char *prefix);\n extern int cmd_verify_commit(int argc, const char **argv, const char *prefix);\n extern int cmd_verify_tag(int argc, const char **argv, const char *prefix);\ndiff --git a/git.c b/git.c\nindex c870b9719..f71073dc8 100644\n--- a/git.c\n+++ b/git.c\n@@ -478,6 +478,7 @@ static struct cmd_struct commands[] = {\n \t{ \"update-server-info\", cmd_update_server_info, RUN_SETUP },\n \t{ \"upload-archive\", cmd_upload_archive },\n \t{ \"upload-archive--writer\", cmd_upload_archive_writer },\n+\t{ \"upload-pack\", cmd_upload_pack },\n \t{ \"var\", cmd_var, RUN_SETUP_GENTLY },\n \t{ \"verify-commit\", cmd_verify_commit, RUN_SETUP },\n \t{ \"verify-pack\", cmd_verify_pack },\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d5de18127..20acaa49d 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1032,7 +1032,7 @@ static int upload_pack_config(const char *var, const char *value, void *unused)\n \treturn parse_hide_refs_config(var, value, \"uploadpack\");\n }\n \n-int cmd_main(int argc, const char **argv)\n+int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n {\n \tconst char *dir;\n \tint strict = 0;\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335655","messageId":"20180103001828.205012-17-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 16/26] ls-remote: pass ref patterns when requesting a remote's refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:18Z","receivedAt":"2018-01-03T00:19:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct an argv_array of the ref patterns supplied via the command\nline and pass them to 'transport_get_remote_refs()' to be used when\ncommunicating protocol v2 so that the server can limit the ref\nadvertisement based on the supplied patterns.\n\nSigned-off-by: Brandon Williams \n---\n builtin/ls-remote.c | 7 +++++--\n t/t5701-protocol-v2.sh | 8 ++++++++\n 2 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex c6e9847c5..caf1051f3 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -43,6 +43,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tint show_symref_target = 0;\n \tconst char *uploadpack = NULL;\n \tconst char **pattern = NULL;\n+\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n \n \tstruct remote *remote;\n \tstruct transport *transport;\n@@ -74,8 +75,10 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (argc > 1) {\n \t\tint i;\n \t\tpattern = xcalloc(argc, sizeof(const char *));\n-\t\tfor (i = 1; i < argc; i++)\n+\t\tfor (i = 1; i < argc; i++) {\n \t\t\tpattern[i - 1] = xstrfmt(\"*/%s\", argv[i]);\n+\t\t\targv_array_push(&ref_patterns, argv[i]);\n+\t\t}\n \t}\n \n \tremote = remote_get(dest);\n@@ -96,7 +99,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (uploadpack != NULL)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n \n-\tref = transport_get_remote_refs(transport, NULL);\n+\tref = transport_get_remote_refs(transport, &ref_patterns);\n \tif (transport_disconnect(transport))\n \t\treturn 1;\n \ndiff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh\nindex 4bf4d61ac..7d8aeb766 100755\n--- a/t/t5701-protocol-v2.sh\n+++ b/t/t5701-protocol-v2.sh\n@@ -25,4 +25,12 @@ test_expect_success 'list refs with file:// using protocol v2' '\n \ttest_cmp actual expect\n '\n \n+test_expect_success 'ref advertisment is filtered with ls-remote using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tls-remote \"file://$(pwd)/file_parent\" master 2>log &&\n+\n+\tgrep \"ref-pattern master\" log &&\n+\t! grep \"refs/tags/\" log\n+'\n+\n test_done\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335656","messageId":"20180103001828.205012-15-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 14/26] transport: convert get_refs_list to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:16Z","receivedAt":"2018-01-03T00:19:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert the 'struct transport' virtual function 'get_refs_list()' to\noptionally take an argv_array of ref patterns. When communicating with\na server using protocol v2 these ref patterns can be sent when\nrequesting a listing of their refs allowing the server to filter the\nrefs it sends based on the sent patterns.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 5 +++--\n transport-internal.h | 4 +++-\n transport.c | 16 +++++++++-------\n 3 files changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 508015023..4c334b5ee 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1026,7 +1026,8 @@ static int has_attribute(const char *attrs, const char *attr) {\n \t}\n }\n \n-static struct ref *get_refs_list(struct transport *transport, int for_push)\n+static struct ref *get_refs_list(struct transport *transport, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns)\n {\n \tstruct helper_data *data = transport->data;\n \tstruct child_process *helper;\n@@ -1039,7 +1040,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)\n \n \tif (process_connect(transport, for_push)) {\n \t\tdo_take_over(transport);\n-\t\treturn transport->vtable->get_refs_list(transport, for_push);\n+\t\treturn transport->vtable->get_refs_list(transport, for_push, ref_patterns);\n \t}\n \n \tif (data->push && for_push)\ndiff --git a/transport-internal.h b/transport-internal.h\nindex 3c1a29d72..a67657ce3 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -3,6 +3,7 @@\n \n struct ref;\n struct transport;\n+struct argv_array;\n \n struct transport_vtable {\n \t/**\n@@ -21,7 +22,8 @@ struct transport_vtable {\n \t * the ref without a huge amount of effort, it should store it\n \t * in the ref's old_sha1 field; otherwise it should be all 0.\n \t **/\n-\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push);\n+\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns);\n \n \t/**\n \t * Fetch the objects for the given refs. Note that this gets\ndiff --git a/transport.c b/transport.c\nindex ffc6b2614..c54a44630 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -72,7 +72,7 @@ struct bundle_transport_data {\n \tstruct bundle_header header;\n };\n \n-static struct ref *get_refs_from_bundle(struct transport *transport, int for_push)\n+static struct ref *get_refs_from_bundle(struct transport *transport, int for_push, const struct argv_array *ref_patterns)\n {\n \tstruct bundle_transport_data *data = transport->data;\n \tstruct ref *result = NULL;\n@@ -189,7 +189,8 @@ static int connect_setup(struct transport *transport, int for_push)\n \treturn 0;\n }\n \n-static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n+static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n+\t\t\t\t\tconst struct argv_array *ref_patterns)\n {\n \tstruct git_transport_data *data = transport->data;\n \tstruct ref *refs = NULL;\n@@ -204,7 +205,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);\n+\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push,\n+\t\t\t\tref_patterns);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n@@ -250,7 +252,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.update_shallow = data->options.update_shallow;\n \n \tif (!data->got_remote_heads)\n-\t\trefs_tmp = get_refs_via_connect(transport, 0);\n+\t\trefs_tmp = get_refs_via_connect(transport, 0, NULL);\n \n \tswitch (data->version) {\n \tcase protocol_v2:\n@@ -568,7 +570,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tint ret = 0;\n \n \tif (!data->got_remote_heads)\n-\t\tget_refs_via_connect(transport, 1);\n+\t\tget_refs_via_connect(transport, 1, NULL);\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n@@ -1028,7 +1030,7 @@ int transport_push(struct transport *transport,\n \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n \t\t\treturn -1;\n \n-\t\tremote_refs = transport->vtable->get_refs_list(transport, 1);\n+\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n \n \t\tif (flags & TRANSPORT_PUSH_ALL)\n \t\t\tmatch_flags |= MATCH_REFS_ALL;\n@@ -1137,7 +1139,7 @@ int transport_push(struct transport *transport,\n const struct ref *transport_get_remote_refs(struct transport *transport)\n {\n \tif (!transport->got_remote_refs) {\n-\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0);\n+\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n \t\ttransport->got_remote_refs = 1;\n \t}\n \n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335657","messageId":"20180103001828.205012-19-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 18/26] push: pass ref patterns when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:20Z","receivedAt":"2018-01-03T00:19:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct a list of ref patterns to be passed to 'get_refs_list()' from\nthe refspec to be used during the push. This list of ref patterns will\nbe used to allow the server to filter the ref advertisement when\ncommunicating using protocol v2.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 17 ++++++++++++++++-\n 1 file changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/transport.c b/transport.c\nindex dfc603b36..6ea3905e3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1026,11 +1026,26 @@ int transport_push(struct transport *transport,\n \t\tint porcelain = flags & TRANSPORT_PUSH_PORCELAIN;\n \t\tint pretend = flags & TRANSPORT_PUSH_DRY_RUN;\n \t\tint push_ret, ret, err;\n+\t\tstruct refspec *tmp_rs;\n+\t\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n+\t\tint i;\n \n \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n \t\t\treturn -1;\n \n-\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n+\t\ttmp_rs = parse_push_refspec(refspec_nr, refspec);\n+\t\tfor (i = 0; i < refspec_nr; i++) {\n+\t\t\tif (tmp_rs[i].dst)\n+\t\t\t\targv_array_push(&ref_patterns, tmp_rs[i].dst);\n+\t\t\telse if (tmp_rs[i].src && !tmp_rs[i].exact_sha1)\n+\t\t\t\targv_array_push(&ref_patterns, tmp_rs[i].src);\n+\t\t}\n+\n+\t\tremote_refs = transport->vtable->get_refs_list(transport, 1,\n+\t\t\t\t\t\t\t &ref_patterns);\n+\n+\t\targv_array_clear(&ref_patterns);\n+\t\tfree_refspec(refspec_nr, tmp_rs);\n \n \t\tif (flags & TRANSPORT_PUSH_ALL)\n \t\t\tmatch_flags |= MATCH_REFS_ALL;\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335658","messageId":"20180103001828.205012-13-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 12/26] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:14Z","receivedAt":"2018-01-03T00:19:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the ls-refs server command. In protocol v2, the ls-refs\ncommand is used to request the ref advertisement from the server. Since\nit is a command which can be requested (as opposed to mandatory in v1),\na client can sent a number of parameters in its request to limit the ref\nadvertisement based on provided ref-patterns.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/technical/protocol-v2.txt | 26 +++++++++\n Makefile | 1 +\n ls-refs.c | 97 +++++++++++++++++++++++++++++++++\n ls-refs.h | 9 +++\n serve.c | 2 +\n 5 files changed, 135 insertions(+)\n create mode 100644 ls-refs.c\n create mode 100644 ls-refs.h\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex b87ba3816..5f4d0e719 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -89,3 +89,29 @@ terminate the connection.\n Commands are the core actions that a client wants to perform (fetch, push,\n etc). Each command will be provided with a list capabilities and\n arguments as requested by a client.\n+\n+ Ls-refs\n+---------\n+\n+Ls-refs is the command used to request a reference advertisement in v2.\n+Unlike the current reference advertisement, ls-refs takes in parameters\n+which can be used to limit the refs sent from the server.\n+\n+Ls-ref takes in the following parameters wraped in packet-lines:\n+\n+ symrefs: In addition to the object pointed by it, show the underlying\n+\t ref pointed by it when showing a symbolic ref.\n+ peel: Show peeled tags.\n+ ref-pattern : When specified, only references matching the\n+\t\t\t given patterns are displayed.\n+\n+The output of ls-refs is as follows:\n+\n+ output = *ref\n+\t flush-pkt\n+ ref = PKT-LINE((tip | peeled) LF)\n+ tip = obj-id SP refname (SP symref-target)\n+ peeled = obj-id SP refname \"^{}\"\n+\n+ symref = PKT-LINE(\"symref\" SP symbolic-ref SP resolved-ref LF)\n+ shallow = PKT-LINE(\"shallow\" SP obj-id LF)\ndiff --git a/Makefile b/Makefile\nindex 5f3b5fe8b..152a73bec 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -820,6 +820,7 @@ LIB_OBJS += list-objects-filter-options.o\n LIB_OBJS += ll-merge.o\n LIB_OBJS += lockfile.o\n LIB_OBJS += log-tree.o\n+LIB_OBJS += ls-refs.o\n LIB_OBJS += mailinfo.o\n LIB_OBJS += mailmap.o\n LIB_OBJS += match-trees.o\ndiff --git a/ls-refs.c b/ls-refs.c\nnew file mode 100644\nindex 000000000..ac4904a40\n--- /dev/null\n+++ b/ls-refs.c\n@@ -0,0 +1,97 @@\n+#include \"cache.h\"\n+#include \"repository.h\"\n+#include \"refs.h\"\n+#include \"remote.h\"\n+#include \"argv-array.h\"\n+#include \"ls-refs.h\"\n+#include \"pkt-line.h\"\n+\n+struct ls_refs_data {\n+\tunsigned peel;\n+\tunsigned symrefs;\n+\tstruct argv_array patterns;\n+};\n+\n+/*\n+ * Check if one of the patterns matches the tail part of the ref.\n+ * If no patterns were provided, all refs match.\n+ */\n+static int ref_match(const struct argv_array *patterns, const char *refname)\n+{\n+\tchar *pathbuf;\n+\tint i;\n+\n+\tif (!patterns->argc)\n+\t\treturn 1; /* no restriction */\n+\n+\tpathbuf = xstrfmt(\"/%s\", refname);\n+\tfor (i = 0; i < patterns->argc; i++) {\n+\t\tif (!wildmatch(patterns->argv[i], pathbuf, 0)) {\n+\t\t\tfree(pathbuf);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\tfree(pathbuf);\n+\treturn 0;\n+}\n+\n+static int send_ref(const char *refname, const struct object_id *oid,\n+\t\t int flag, void *cb_data)\n+{\n+\tstruct ls_refs_data *data = cb_data;\n+\tconst char *refname_nons = strip_namespace(refname);\n+\tstruct strbuf refline = STRBUF_INIT;\n+\n+\tif (!ref_match(&data->patterns, refname))\n+\t\treturn 0;\n+\n+\tstrbuf_addf(&refline, \"%s %s\", oid_to_hex(oid), refname_nons);\n+\tif (data->symrefs && flag & REF_ISSYMREF) {\n+\t\tstruct object_id unused;\n+\t\tconst char *symref_target = resolve_ref_unsafe(refname, 0,\n+\t\t\t\t\t\t\t &unused,\n+\t\t\t\t\t\t\t &flag);\n+\n+\t\tif (!symref_target)\n+\t\t\tdie(\"'%s' is a symref but it is not?\", refname);\n+\n+\t\tstrbuf_addf(&refline, \" %s\", symref_target);\n+\t}\n+\n+\tstrbuf_addch(&refline, '\\n');\n+\n+\tpacket_write(1, refline.buf, refline.len);\n+\tif (data->peel) {\n+\t\tstruct object_id peeled;\n+\t\tif (!peel_ref(refname, &peeled))\n+\t\t\tpacket_write_fmt(1, \"%s %s^{}\\n\", oid_to_hex(&peeled),\n+\t\t\t\t\t refname_nons);\n+\t}\n+\n+\tstrbuf_release(&refline);\n+\treturn 0;\n+}\n+\n+int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)\n+{\n+\tint i;\n+\tstruct ls_refs_data data = { 0, 0, ARGV_ARRAY_INIT };\n+\n+\tfor (i = 0; i < args->argc; i++) {\n+\t\tconst char *arg = args->argv[i];\n+\t\tconst char *out;\n+\n+\t\tif (!strcmp(\"peel\", arg))\n+\t\t\tdata.peel = 1;\n+\t\telse if (!strcmp(\"symrefs\", arg))\n+\t\t\tdata.symrefs = 1;\n+\t\telse if (skip_prefix(arg, \"ref-pattern \", &out))\n+\t\t\targv_array_pushf(&data.patterns, \"*/%s\", out);\n+\t}\n+\n+\thead_ref_namespaced(send_ref, &data);\n+\tfor_each_namespaced_ref(send_ref, &data);\n+\tpacket_flush(1);\n+\targv_array_clear(&data.patterns);\n+\treturn 0;\n+}\ndiff --git a/ls-refs.h b/ls-refs.h\nnew file mode 100644\nindex 000000000..9e4c57bfe\n--- /dev/null\n+++ b/ls-refs.h\n@@ -0,0 +1,9 @@\n+#ifndef LS_REFS_H\n+#define LS_REFS_H\n+\n+struct repository;\n+struct argv_array;\n+extern int ls_refs(struct repository *r, struct argv_array *keys,\n+\t\t struct argv_array *args);\n+\n+#endif /* LS_REFS_H */\ndiff --git a/serve.c b/serve.c\nindex da8127775..88d548410 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -4,6 +4,7 @@\n #include \"pkt-line.h\"\n #include \"version.h\"\n #include \"argv-array.h\"\n+#include \"ls-refs.h\"\n #include \"serve.h\"\n \n static int always_advertise(struct repository *r,\n@@ -44,6 +45,7 @@ struct protocol_capability {\n static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n \t{ \"stateless-rpc\", always_advertise, NULL },\n+\t{ \"ls-refs\", always_advertise, ls_refs },\n };\n \n static void advertise_capabilities(void)\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335659","messageId":"20180103001828.205012-26-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 25/26] remote-curl: create copy of the service name","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:27Z","receivedAt":"2018-01-03T00:19:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Make a copy of the service name being requested instead of relying on\nthe buffer pointed to by the passed in 'const char *' to remain\nunchanged.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex dae8a4a48..4086aa733 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -165,7 +165,7 @@ static int set_option(const char *name, const char *value)\n }\n \n struct discovery {\n-\tconst char *service;\n+\tchar *service;\n \tchar *buf_alloc;\n \tchar *buf;\n \tsize_t len;\n@@ -257,6 +257,7 @@ static void free_discovery(struct discovery *d)\n \t\tfree(d->shallow.oid);\n \t\tfree(d->buf_alloc);\n \t\tfree_refs(d->refs);\n+\t\tfree(d->service);\n \t\tfree(d);\n \t}\n }\n@@ -343,7 +344,7 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \t\twarning(_(\"redirecting to %s\"), url.buf);\n \n \tlast= xcalloc(1, sizeof(*last_discovery));\n-\tlast->service = service;\n+\tlast->service = xstrdup(service);\n \tlast->buf_alloc = strbuf_detach(&buffer, &last->len);\n \tlast->buf = last->buf_alloc;\n \n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335660","messageId":"20180103001828.205012-27-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 26/26] remote-curl: implement connect-half-duplex command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:28Z","receivedAt":"2018-01-03T00:19:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach remote-curl the 'connect-half-duplex' command which is used to\nestablish a half-duplex connection with servers which support protocol\nversion 2. This allows remote-curl to act as a proxy, allowing the git\nclient to communicate natively with a remote end, simply using\nremote-curl as a pass through to convert requests to http.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 185 ++++++++++++++++++++++++++++++++++++++++++++++++-\n t/t5701-protocol-v2.sh | 41 +++++++++++\n 2 files changed, 224 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 4086aa733..b63b06398 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -171,6 +171,7 @@ struct discovery {\n \tsize_t len;\n \tstruct ref *refs;\n \tstruct oid_array shallow;\n+\tenum protocol_version version;\n \tunsigned proto_git : 1;\n };\n static struct discovery *last_discovery;\n@@ -184,9 +185,13 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n-\tswitch (discover_version(&reader)) {\n+\theads->version = discover_version(&reader);\n+\tswitch (heads->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\t/*\n+\t\t * Do nothing. Client should run 'connect-half-duplex' and\n+\t\t * request the refs themselves.\n+\t\t */\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n@@ -1047,6 +1052,178 @@ static void parse_push(struct strbuf *buf)\n \tfree(specs);\n }\n \n+struct proxy_state {\n+\tchar *service_name;\n+\tchar *service_url;\n+\tchar *hdr_content_type;\n+\tchar *hdr_accept;\n+\tstruct strbuf request_buffer;\n+\tint in;\n+\tint out;\n+\tstruct packet_reader reader;\n+\tsize_t pos;\n+\tint seen_flush;\n+};\n+\n+static void proxy_state_init(struct proxy_state *p, const char *service_name)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tmemset(p, 0, sizeof(*p));\n+\tp->service_name = xstrdup(service_name);\n+\n+\tp->in = 0;\n+\tp->out = 1;\n+\tstrbuf_init(&p->request_buffer, 0);\n+\n+\tstrbuf_addf(&buf, \"%s%s\", url.buf, p->service_name);\n+\tp->service_url = strbuf_detach(&buf, NULL);\n+\n+\tstrbuf_addf(&buf, \"Content-Type: application/x-%s-request\", p->service_name);\n+\tp->hdr_content_type = strbuf_detach(&buf, NULL);\n+\n+\tstrbuf_addf(&buf, \"Accept: application/x-%s-result\", p->service_name);\n+\tp->hdr_accept = strbuf_detach(&buf, NULL);\n+\n+\tpacket_reader_init(&p->reader, p->in, NULL, 0,\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+}\n+\n+static void proxy_state_clear(struct proxy_state *p)\n+{\n+\tfree(p->service_name);\n+\tfree(p->service_url);\n+\tfree(p->hdr_content_type);\n+\tfree(p->hdr_accept);\n+\tstrbuf_release(&p->request_buffer);\n+}\n+\n+static size_t proxy_in(void *ptr, size_t eltsize,\n+\t\t size_t nmemb, void *buffer_)\n+{\n+\tsize_t max = eltsize * nmemb;\n+\tstruct proxy_state *p = buffer_;\n+\tsize_t avail = p->request_buffer.len - p->pos;\n+\n+\tif (!avail) {\n+\t\tif (p->seen_flush) {\n+\t\t\tp->seen_flush = 0;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tstrbuf_reset(&p->request_buffer);\n+\t\tswitch (packet_reader_read(&p->reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tdie(\"error reading request from parent process\");\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tpacket_buf_write_len(&p->request_buffer, p->reader.line,\n+\t\t\t\t\t p->reader.pktlen);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tpacket_buf_delim(&p->request_buffer);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tpacket_buf_flush(&p->request_buffer);\n+\t\t\tp->seen_flush = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tp->pos = 0;\n+\t\tavail = p->request_buffer.len;\n+\t}\n+\n+\tif (max < avail)\n+\t\tavail = max;\n+\tmemcpy(ptr, p->request_buffer.buf + p->pos, avail);\n+\tp->pos += avail;\n+\treturn avail;\n+}\n+static size_t proxy_out(char *ptr, size_t eltsize,\n+\t\t\tsize_t nmemb, void *buffer_)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct proxy_state *p = buffer_;\n+\n+\twrite_or_die(p->out, ptr, size);\n+\treturn size;\n+}\n+\n+static int proxy_post(struct proxy_state *p)\n+{\n+\tstruct active_request_slot *slot;\n+\tstruct curl_slist *headers = http_copy_default_headers();\n+\tint err;\n+\n+\theaders = curl_slist_append(headers, p->hdr_content_type);\n+\theaders = curl_slist_append(headers, p->hdr_accept);\n+\theaders = curl_slist_append(headers, \"Transfer-Encoding: chunked\");\n+\n+\tslot = get_active_slot();\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n+\n+\t/* Setup function to read request from client */\n+\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_READDATA, p);\n+\n+\t/* Setup function to write server response to client */\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);\n+\n+\terr = run_slot(slot, NULL);\n+\n+\tif (err != HTTP_OK)\n+\t\terr = -1;\n+\n+\tcurl_slist_free_all(headers);\n+\treturn err;\n+}\n+\n+static int connect_half_duplex(const char *service_name)\n+{\n+\tstruct discovery *discover;\n+\tstruct proxy_state p;\n+\n+\t/*\n+\t * Run the info/refs request and see if the server supports protocol\n+\t * v2. If and only if the server supports v2 can we successfully\n+\t * establish a half-duplex connection, otherwise we need to tell the\n+\t * client to fallback to using other transport helper functions to\n+\t * complete their request.\n+\t */\n+\tdiscover = discover_refs(service_name, 0);\n+\tif (discover->version != protocol_v2) {\n+\t\tprintf(\"fallback\\n\");\n+\t\tfflush(stdout);\n+\t\treturn -1;\n+\t} else {\n+\t\t/* Half-Duplex Connection established */\n+\t\tprintf(\"\\n\");\n+\t\tfflush(stdout);\n+\t}\n+\n+\tproxy_state_init(&p, service_name);\n+\n+\t/*\n+\t * Dump the capability listing that we got from the server earlier\n+\t * during the info/refs request.\n+\t */\n+\twrite_or_die(p.out, discover->buf, discover->len);\n+\n+\t/* Peek the next packet line. Until we see EOF keep sending POSTs */\n+\twhile (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {\n+\t\tif (proxy_post(&p)) {\n+\t\t\t/* We would have an err here */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tproxy_state_clear(&p);\n+\treturn 0;\n+}\n+\n int cmd_main(int argc, const char **argv)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -1115,12 +1292,16 @@ int cmd_main(int argc, const char **argv)\n \t\t\tfflush(stdout);\n \n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n+\t\t\tprintf(\"connect-half-duplex\\n\");\n \t\t\tprintf(\"fetch\\n\");\n \t\t\tprintf(\"option\\n\");\n \t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"check-connectivity\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n+\t\t} else if (skip_prefix(buf.buf, \"connect-half-duplex \", &arg)) {\n+\t\t\tif (!connect_half_duplex(arg))\n+\t\t\t\tbreak;\n \t\t} else {\n \t\t\terror(\"remote-curl: unknown command '%s' from git\", buf.buf);\n \t\t\treturn 1;\ndiff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh\nindex 3e411e178..ada69ac09 100755\n--- a/t/t5701-protocol-v2.sh\n+++ b/t/t5701-protocol-v2.sh\n@@ -73,4 +73,45 @@ test_expect_success 'ref advertisment is filtered during fetch using protocol v2\n \t! grep \"refs/tags/\" log\n '\n \n+# Test protocol v2 with 'http://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one\n+'\n+\n+test_expect_success 'clone with http:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 GIT_TRACE_CURL=1 git -c protocol.version=2 \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"Git-Protocol: version=2\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log\n+'\n+\n+test_expect_success 'fetch with http:// using protocol v2' '\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=2 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335661","messageId":"20180103001828.205012-23-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 22/26] transport-helper: refactor process_connect_service","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:24Z","receivedAt":"2018-01-03T00:19:30Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"A future patch will need to take advantage of the logic which runs and\nprocesses the response of the connect command on a remote helper so\nfactor out this logic from 'process_connect_service()' and place it into\na helper function 'run_connect()'.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 67 +++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 38 insertions(+), 29 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex d72155768..c032a2a87 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -545,14 +545,13 @@ static int fetch_with_import(struct transport *transport,\n \treturn 0;\n }\n \n-static int process_connect_service(struct transport *transport,\n-\t\t\t\t const char *name, const char *exec)\n+static int run_connect(struct transport *transport, struct strbuf *cmdbuf)\n {\n \tstruct helper_data *data = transport->data;\n-\tstruct strbuf cmdbuf = STRBUF_INIT;\n-\tstruct child_process *helper;\n-\tint r, duped, ret = 0;\n+\tint ret = 0;\n+\tint duped;\n \tFILE *input;\n+\tstruct child_process *helper;\n \n \thelper = get_helper(transport);\n \n@@ -568,44 +567,54 @@ static int process_connect_service(struct transport *transport,\n \tinput = xfdopen(duped, \"r\");\n \tsetvbuf(input, NULL, _IONBF, 0);\n \n+\tsendline(data, cmdbuf);\n+\tif (recvline_fh(input, cmdbuf))\n+\t\texit(128);\n+\n+\tif (!strcmp(cmdbuf->buf, \"\")) {\n+\t\tdata->no_disconnect_req = 1;\n+\t\tif (debug)\n+\t\t\tfprintf(stderr, \"Debug: Smart transport connection \"\n+\t\t\t\t\"ready.\\n\");\n+\t\tret = 1;\n+\t} else if (!strcmp(cmdbuf->buf, \"fallback\")) {\n+\t\tif (debug)\n+\t\t\tfprintf(stderr, \"Debug: Falling back to dumb \"\n+\t\t\t\t\"transport.\\n\");\n+\t} else {\n+\t\tdie(\"Unknown response to connect: %s\",\n+\t\t\tcmdbuf->buf);\n+\t}\n+\n+\tfclose(input);\n+\treturn ret;\n+}\n+\n+static int process_connect_service(struct transport *transport,\n+\t\t\t\t const char *name, const char *exec)\n+{\n+\tstruct helper_data *data = transport->data;\n+\tstruct strbuf cmdbuf = STRBUF_INIT;\n+\tint ret = 0;\n+\n \t/*\n \t * Handle --upload-pack and friends. This is fire and forget...\n \t * just warn if it fails.\n \t */\n \tif (strcmp(name, exec)) {\n-\t\tr = set_helper_option(transport, \"servpath\", exec);\n+\t\tint r = set_helper_option(transport, \"servpath\", exec);\n \t\tif (r > 0)\n \t\t\twarning(\"Setting remote service path not supported by protocol.\");\n \t\telse if (r < 0)\n \t\t\twarning(\"Invalid remote service path.\");\n \t}\n \n-\tif (data->connect)\n+\tif (data->connect) {\n \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n-\telse\n-\t\tgoto exit;\n-\n-\tsendline(data, &cmdbuf);\n-\tif (recvline_fh(input, &cmdbuf))\n-\t\texit(128);\n-\n-\tif (!strcmp(cmdbuf.buf, \"\")) {\n-\t\tdata->no_disconnect_req = 1;\n-\t\tif (debug)\n-\t\t\tfprintf(stderr, \"Debug: Smart transport connection \"\n-\t\t\t\t\"ready.\\n\");\n-\t\tret = 1;\n-\t} else if (!strcmp(cmdbuf.buf, \"fallback\")) {\n-\t\tif (debug)\n-\t\t\tfprintf(stderr, \"Debug: Falling back to dumb \"\n-\t\t\t\t\"transport.\\n\");\n-\t} else\n-\t\tdie(\"Unknown response to connect: %s\",\n-\t\t\tcmdbuf.buf);\n+\t\tret = run_connect(transport, &cmdbuf);\n+\t}\n \n-exit:\n \tstrbuf_release(&cmdbuf);\n-\tfclose(input);\n \treturn ret;\n }\n \n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335662","messageId":"20180103001828.205012-21-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 20/26] fetch-pack: perform a fetch using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:22Z","receivedAt":"2018-01-03T00:19:32Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"When communicating with a v2 server, perform a fetch by requesting the\n'fetch' command.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 2 +-\n fetch-pack.c | 267 ++++++++++++++++++++++++++++++++++++++++++++++++-\n fetch-pack.h | 4 +-\n t/t5701-protocol-v2.sh | 40 ++++++++\n transport.c | 8 +-\n 5 files changed, 314 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex f492e8abd..867dd3cc7 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -213,7 +213,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n-\t\t\t &shallow, pack_lockfile_ptr);\n+\t\t\t &shallow, pack_lockfile_ptr, protocol_v0);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 9f6b07ad9..c26fdc539 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1008,6 +1008,262 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \treturn ref;\n }\n \n+static void add_wants(const struct ref *wants, struct strbuf *req_buf)\n+{\n+\tfor ( ; wants ; wants = wants->next) {\n+\t\tconst struct object_id *remote = &wants->old_oid;\n+\t\tconst char *remote_hex;\n+\t\tstruct object *o;\n+\n+\t\t/*\n+\t\t * If that object is complete (i.e. it is an ancestor of a\n+\t\t * local ref), we tell them we have it but do not have to\n+\t\t * tell them about its ancestors, which they already know\n+\t\t * about.\n+\t\t *\n+\t\t * We use lookup_object here because we are only\n+\t\t * interested in the case we *know* the object is\n+\t\t * reachable and we have already scanned it.\n+\t\t */\n+\t\tif (((o = lookup_object(remote->hash)) != NULL) &&\n+\t\t (o->flags & COMPLETE)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tremote_hex = oid_to_hex(remote);\n+\t\tpacket_buf_write(req_buf, \"want %s\\n\", remote_hex);\n+\t}\n+}\n+\n+static int add_haves(struct strbuf *req_buf, int *in_vain)\n+{\n+\tint ret = 0;\n+\tint haves_added = 0;\n+\tconst struct object_id *oid;\n+\n+\twhile ((oid = get_rev())) {\n+\t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n+\t\tif (++haves_added >= INITIAL_FLUSH)\n+\t\t\tbreak;\n+\t};\n+\n+\t*in_vain += haves_added;\n+\tif (!haves_added || *in_vain >= MAX_IN_VAIN) {\n+\t\t/* Send Done */\n+\t\tpacket_buf_write(req_buf, \"done\\n\");\n+\t\tret = 1;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static int send_haves(int fd_out, int *in_vain)\n+{\n+\tint ret = 0;\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\n+\tret = add_haves(&req_buf, in_vain);\n+\n+\t/* Send request */\n+\tpacket_buf_flush(&req_buf);\n+\twrite_or_die(fd_out, req_buf.buf, req_buf.len);\n+\n+\tstrbuf_release(&req_buf);\n+\treturn ret;\n+}\n+\n+static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n+\t\t\t const struct ref *wants, struct oidset *common,\n+\t\t\t int *in_vain)\n+{\n+\tint ret = 0;\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\n+\tpacket_buf_write(&req_buf, \"command=fetch\");\n+\tpacket_buf_write(&req_buf, \"agent=%s\", git_user_agent_sanitized());\n+\tif (args->stateless_rpc)\n+\t\tpacket_buf_write(&req_buf, \"stateless-rpc=true\");\n+\n+\tpacket_buf_delim(&req_buf);\n+\tif (args->use_thin_pack)\n+\t\tpacket_buf_write(&req_buf, \"thin-pack\");\n+\tif (args->no_progress)\n+\t\tpacket_buf_write(&req_buf, \"no-progress\");\n+\tif (args->include_tag)\n+\t\tpacket_buf_write(&req_buf, \"include-tag\");\n+\tif (prefer_ofs_delta)\n+\t\tpacket_buf_write(&req_buf, \"ofs-delta\");\n+\n+\t/* add wants */\n+\tadd_wants(wants, &req_buf);\n+\n+\t/*\n+\t * If we are running stateless-rpc we need to add all the common\n+\t * commits we've found in previous rounds\n+\t */\n+\tif (args->stateless_rpc) {\n+\t\tstruct oidset_iter iter;\n+\t\tconst struct object_id *oid;\n+\t\toidset_iter_init(common, &iter);\n+\n+\t\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n+\t\t}\n+\t}\n+\n+\t/* Add initial haves */\n+\tret = add_haves(&req_buf, in_vain);\n+\n+\t/* Send request */\n+\tpacket_buf_flush(&req_buf);\n+\twrite_or_die(fd_out, req_buf.buf, req_buf.len);\n+\n+\tstrbuf_release(&req_buf);\n+\treturn ret;\n+}\n+\n+static enum ack_type process_ack(const char *line, struct object_id *oid)\n+{\n+\tconst char *arg;\n+\n+\tif (!strcmp(line, \"NAK\"))\n+\t\treturn NAK;\n+\tif (skip_prefix(line, \"ACK \", &arg)) {\n+\t\tif (!parse_oid_hex(arg, oid, &arg)) {\n+\t\t\tif (strstr(arg, \"continue\"))\n+\t\t\t\treturn ACK_continue;\n+\t\t\tif (strstr(arg, \"common\"))\n+\t\t\t\treturn ACK_common;\n+\t\t\tif (strstr(arg, \"ready\"))\n+\t\t\t\treturn ACK_ready;\n+\t\t\treturn ACK;\n+\t\t}\n+\t}\n+\tif (skip_prefix(line, \"ERR \", &arg))\n+\t\tdie(_(\"remote error: %s\"), arg);\n+\tdie(_(\"git fetch-pack: expected ACK/NAK, got '%s'\"), line);\n+}\n+\n+static int process_acks(struct packet_reader *reader, struct oidset *common)\n+{\n+\tint got_ready = 0;\n+\tint got_common = 0;\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tstruct object_id oid;\n+\t\tstruct commit *commit;\n+\t\tenum ack_type ack = process_ack(reader->line, &oid);\n+\n+\t\tswitch (ack) {\n+\t\tcase ACK_ready:\n+\t\t\tclear_prio_queue(&rev_list);\n+\t\t\tgot_ready = 1;\n+\t\t\t/* fallthrough */\n+\t\tcase ACK_common:\n+\t\t\toidset_insert(common, &oid);\n+\t\t\tcommit = lookup_commit(&oid);\n+\t\t\tmark_common(commit, 0, 1);\n+\t\t\tgot_common = 1;\n+\t\t\tbreak;\n+\t\tcase NAK:\n+\t\t\tbreak;\n+\t\tcase ACK:\n+\t\tcase ACK_continue:\n+\t\t\tdie(\"ACK/ACK_continue not supported\");\n+\t\t}\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH &&\n+\t reader->status != PACKET_READ_DELIM)\n+\t\tdie(\"Error during processing acks: %d\", reader->status);\n+\n+\t/* return 0 if no common, 1 if there are common, or 2 if ready */\n+\treturn got_ready + got_common;\n+}\n+\n+#define FETCH_CHECK_LOCAL 0\n+#define FETCH_SEND_REQUEST 1\n+#define FETCH_PROCESS_ACKS 2\n+#define FETCH_SEND_HAVES 3\n+#define FETCH_GET_PACK 4\n+#define FETCH_DONE 5\n+\n+static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n+\t\t\t\t int fd[2],\n+\t\t\t\t const struct ref *orig_ref,\n+\t\t\t\t struct ref **sought, int nr_sought,\n+\t\t\t\t char **pack_lockfile)\n+{\n+\tstruct ref *ref = copy_ref_list(orig_ref);\n+\tint state = FETCH_CHECK_LOCAL;\n+\tstruct oidset common = OIDSET_INIT;\n+\tstruct packet_reader reader;\n+\tint in_vain = 0;\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (state != FETCH_DONE) {\n+\t\tswitch (state) {\n+\t\tcase FETCH_CHECK_LOCAL:\n+\t\t\tsort_ref_list(&ref, ref_compare_name);\n+\t\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n+\n+\t\t\t/* v2 supports these by default */\n+\t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n+\t\t\tuse_sideband = 2;\n+\n+\t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n+\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_SEND_REQUEST;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_REQUEST:\n+\t\t\tif (send_fetch_request(fd[1], args, ref, &common, &in_vain))\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_PROCESS_ACKS;\n+\t\t\tbreak;\n+\t\tcase FETCH_PROCESS_ACKS:\n+\t\t\t/* Process ACKs/NAKs */\n+\t\t\tswitch (process_acks(&reader, &common)) {\n+\t\t\tcase 2:\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\t\tbreak;\n+\t\t\tcase 1:\n+\t\t\t\tin_vain = 0;\n+\t\t\t\t/* fallthrough */\n+\t\t\tdefault:\n+\t\t\t\tif (args->stateless_rpc)\n+\t\t\t\t\tstate = FETCH_SEND_REQUEST;\n+\t\t\t\telse\n+\t\t\t\t\tstate = FETCH_SEND_HAVES;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_HAVES:\n+\t\t\tif (send_haves(fd[1], &in_vain))\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_PROCESS_ACKS;\n+\t\t\tbreak;\n+\t\tcase FETCH_GET_PACK:\n+\t\t\t/* get the pack */\n+\t\t\tif (get_pack(args, fd, pack_lockfile))\n+\t\t\t\tdie(_(\"git fetch-pack: fetch failed.\"));\n+\n+\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_DONE:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tdie(\"invalid state\");\n+\t\t}\n+\t}\n+\n+\toidset_clear(&common);\n+\treturn ref;\n+}\n+\n static void fetch_pack_config(void)\n {\n \tgit_config_get_int(\"fetch.unpacklimit\", &fetch_unpack_limit);\n@@ -1153,7 +1409,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const char *dest,\n \t\t struct ref **sought, int nr_sought,\n \t\t struct oid_array *shallow,\n-\t\t char **pack_lockfile)\n+\t\t char **pack_lockfile,\n+\t\t enum protocol_version version)\n {\n \tstruct ref *ref_cpy;\n \tstruct shallow_info si;\n@@ -1167,8 +1424,12 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tdie(_(\"no matching remote head\"));\n \t}\n \tprepare_shallow_info(&si, shallow);\n-\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n-\t\t\t\t&si, pack_lockfile);\n+\tif (version == protocol_v2)\n+\t\tref_cpy = do_fetch_pack_v2(args, fd, ref, sought, nr_sought,\n+\t\t\t\t\t pack_lockfile);\n+\telse\n+\t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n+\t\t\t\t\t&si, pack_lockfile);\n \treprepare_packed_git();\n \tupdate_shallow(args, sought, nr_sought, &si);\n \tclear_shallow_info(&si);\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex b6aeb43a8..7afca7305 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -3,6 +3,7 @@\n \n #include \"string-list.h\"\n #include \"run-command.h\"\n+#include \"protocol.h\"\n \n struct oid_array;\n \n@@ -43,7 +44,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t struct ref **sought,\n \t\t int nr_sought,\n \t\t struct oid_array *shallow,\n-\t\t char **pack_lockfile);\n+\t\t char **pack_lockfile,\n+\t\t enum protocol_version version);\n \n /*\n * Print an appropriate error message for each sought ref that wasn't\ndiff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh\nindex 7d8aeb766..3e411e178 100755\n--- a/t/t5701-protocol-v2.sh\n+++ b/t/t5701-protocol-v2.sh\n@@ -33,4 +33,44 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol\n \t! grep \"refs/tags/\" log\n '\n \n+test_expect_success 'clone with file:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tclone \"file://$(pwd)/file_parent\" file_child 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 2\" log\n+'\n+\n+test_expect_success 'fetch with file:// using protocol v2' '\n+\ttest_commit -C file_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n+\t\tfetch origin 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 2\" log\n+'\n+\n+test_expect_success 'ref advertisment is filtered during fetch using protocol v2' '\n+\ttest_commit -C file_parent three &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n+\t\tfetch origin master 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgrep \"ref-pattern master\" log &&\n+\t! grep \"refs/tags/\" log\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex 6ea3905e3..4fdbd9adc 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -256,14 +256,18 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\trefs = fetch_pack(&args, data->fd, data->conn,\n+\t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n+\t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n+\t\t\t\t &transport->pack_lockfile, data->version);\n+\t\tpacket_flush(data->fd[1]);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n \t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n-\t\t\t\t &transport->pack_lockfile);\n+\t\t\t\t &transport->pack_lockfile, data->version);\n \t\tbreak;\n \tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335663","messageId":"20180103001828.205012-22-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 21/26] transport-helper: remove name parameter","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:23Z","receivedAt":"2018-01-03T00:19:34Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Commit 266f1fdfa (transport-helper: be quiet on read errors from\nhelpers, 2013-06-21) removed a call to 'die()' which printed the name of\nthe remote helper passed in to the 'recvline_fh()' function using the\n'name' parameter. Once the call to 'die()' was removed the parameter\nwas no longer necessary but wasn't removed. Clean up 'recvline_fh()'\nparameter list by removing the 'name' parameter.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 4c334b5ee..d72155768 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -49,7 +49,7 @@ static void sendline(struct helper_data *helper, struct strbuf *buffer)\n \t\tdie_errno(\"Full write to remote helper failed\");\n }\n \n-static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)\n+static int recvline_fh(FILE *helper, struct strbuf *buffer)\n {\n \tstrbuf_reset(buffer);\n \tif (debug)\n@@ -67,7 +67,7 @@ static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)\n \n static int recvline(struct helper_data *helper, struct strbuf *buffer)\n {\n-\treturn recvline_fh(helper->out, buffer, helper->name);\n+\treturn recvline_fh(helper->out, buffer);\n }\n \n static void write_constant(int fd, const char *str)\n@@ -586,7 +586,7 @@ static int process_connect_service(struct transport *transport,\n \t\tgoto exit;\n \n \tsendline(data, &cmdbuf);\n-\tif (recvline_fh(input, &cmdbuf, name))\n+\tif (recvline_fh(input, &cmdbuf))\n \t\texit(128);\n \n \tif (!strcmp(cmdbuf.buf, \"\")) {\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335664","messageId":"20180103001828.205012-24-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 23/26] transport-helper: introduce connect-half-duplex","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:25Z","receivedAt":"2018-01-03T00:19:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the transport-helper capability 'connect-half-duplex'. This\ncapability indicates that the transport-helper can be requested to run\nthe 'connect-half-duplex' command which should attempt to make a\nhalf-duplex connection with a remote end. Once established, the\nhalf-duplex connection can be used by the git client to communicate with\nthe remote end natively in a stateles-rpc manner as supported by\nprotocol v2.\n\nIf a half-duplex connection cannot be established then the remote-helper\nwill respond in the same manner as the 'connect' command indicating that\nthe client should fallback to using the dumb remote-helper commands.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 8 ++++++++\n transport.c | 1 +\n transport.h | 6 ++++++\n 3 files changed, 15 insertions(+)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c032a2a87..d037609bc 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -26,6 +26,7 @@ struct helper_data {\n \t\toption : 1,\n \t\tpush : 1,\n \t\tconnect : 1,\n+\t\tconnect_half_duplex : 1,\n \t\tsigned_tags : 1,\n \t\tcheck_connectivity : 1,\n \t\tno_disconnect_req : 1,\n@@ -188,6 +189,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\trefspecs[refspec_nr++] = xstrdup(arg);\n \t\t} else if (!strcmp(capname, \"connect\")) {\n \t\t\tdata->connect = 1;\n+\t\t} else if (!strcmp(capname, \"connect-half-duplex\")) {\n+\t\t\tdata->connect_half_duplex = 1;\n \t\t} else if (!strcmp(capname, \"signed-tags\")) {\n \t\t\tdata->signed_tags = 1;\n \t\t} else if (skip_prefix(capname, \"export-marks \", &arg)) {\n@@ -612,6 +615,11 @@ static int process_connect_service(struct transport *transport,\n \tif (data->connect) {\n \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n \t\tret = run_connect(transport, &cmdbuf);\n+\t} else if (data->connect_half_duplex) {\n+\t\tstrbuf_addf(&cmdbuf, \"connect-half-duplex %s\\n\", name);\n+\t\tret = run_connect(transport, &cmdbuf);\n+\t\tif (ret)\n+\t\t\ttransport->stateless_rpc = 1;\n \t}\n \n \tstrbuf_release(&cmdbuf);\ndiff --git a/transport.c b/transport.c\nindex 4fdbd9adc..aafb8fbb4 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -250,6 +250,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n+\targs.stateless_rpc = transport->stateless_rpc;\n \n \tif (!data->got_remote_heads)\n \t\trefs_tmp = get_refs_via_connect(transport, 0, NULL);\ndiff --git a/transport.h b/transport.h\nindex 4b656f315..9eac809ee 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -55,6 +55,12 @@ struct transport {\n \t */\n \tunsigned cloning : 1;\n \n+\t/*\n+\t * Indicates that the transport is connected via a half-duplex\n+\t * connection and should operate in stateless-rpc mode.\n+\t */\n+\tunsigned stateless_rpc : 1;\n+\n \t/*\n \t * These strings will be passed to the {pre, post}-receive hook,\n \t * on the remote side, if both sides support the push options capability.\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335665","messageId":"20180103001828.205012-20-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 19/26] upload-pack: introduce fetch server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:21Z","receivedAt":"2018-01-03T00:19:41Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the 'fetch' server command.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/technical/protocol-v2.txt | 14 ++\n serve.c | 2 +\n upload-pack.c | 290 ++++++++++++++++++++++++++++++++\n upload-pack.h | 9 +\n 4 files changed, 315 insertions(+)\n create mode 100644 upload-pack.h\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 5f4d0e719..2a8e2f226 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -115,3 +115,17 @@ The output of ls-refs is as follows:\n \n symref = PKT-LINE(\"symref\" SP symbolic-ref SP resolved-ref LF)\n shallow = PKT-LINE(\"shallow\" SP obj-id LF)\n+\n+ Fetch\n+-------\n+\n+Fetch will need to be a modified version of the v1 fetch protocol. Some\n+potential areas for improvement are: Ref-in-want, CDN offloading,\n+Fetch-options.\n+\n+Since we'll have an 'ls-ref' service we can eliminate the need of fetch\n+to perform a ref-advertisement, instead a client can run the 'ls-refs'\n+service first, in order to find out what refs the server has, and then\n+request those refs directly using the fetch service.\n+\n+//TODO Flesh out the design\ndiff --git a/serve.c b/serve.c\nindex 88d548410..ca3bb7190 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -6,6 +6,7 @@\n #include \"argv-array.h\"\n #include \"ls-refs.h\"\n #include \"serve.h\"\n+#include \"upload-pack.h\"\n \n static int always_advertise(struct repository *r,\n \t\t\t struct strbuf *value)\n@@ -46,6 +47,7 @@ static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n \t{ \"stateless-rpc\", always_advertise, NULL },\n \t{ \"ls-refs\", always_advertise, ls_refs },\n+\t{ \"fetch\", always_advertise, upload_pack_v2 },\n };\n \n static void advertise_capabilities(void)\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 2ca60d27c..c41f6f528 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -20,6 +20,7 @@\n #include \"prio-queue.h\"\n #include \"protocol.h\"\n #include \"serve.h\"\n+#include \"upload-pack.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -1040,6 +1041,295 @@ static void upload_pack(void)\n \t}\n }\n \n+struct upload_pack_data {\n+\tstruct object_array wants;\n+\tstruct oid_array haves;\n+\n+\tunsigned stateless_rpc : 1;\n+\n+\tunsigned use_thin_pack : 1;\n+\tunsigned use_ofs_delta : 1;\n+\tunsigned no_progress : 1;\n+\tunsigned use_include_tag : 1;\n+\tunsigned done : 1;\n+};\n+\n+#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }\n+\n+static void upload_pack_data_clear(struct upload_pack_data *data)\n+{\n+\tobject_array_clear(&data->wants);\n+\toid_array_clear(&data->haves);\n+}\n+\n+static int parse_want(const char *line)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"want \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tstruct object *o;\n+\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"git upload-pack: protocol error, \"\n+\t\t\t \"expected to get oid, not '%s'\", line);\n+\n+\t\to = parse_object(&oid);\n+\t\tif (!o) {\n+\t\t\tpacket_write_fmt(1,\n+\t\t\t\t\t \"ERR upload-pack: not our ref %s\",\n+\t\t\t\t\t oid_to_hex(&oid));\n+\t\t\tdie(\"git upload-pack: not our ref %s\",\n+\t\t\t oid_to_hex(&oid));\n+\t\t}\n+\n+\t\tif (!(o->flags & WANTED)) {\n+\t\t\to->flags |= WANTED;\n+\t\t\tadd_object_array(o, NULL, &want_obj);\n+\t\t}\n+\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int parse_have(const char *line, struct oid_array *haves)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"have \", &arg)) {\n+\t\tstruct object_id oid;\n+\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"git upload-pack: expected SHA1 object, got '%s'\", arg);\n+\t\toid_array_append(haves, &oid);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void process_args(struct argv_array *args, struct upload_pack_data *data)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < args->argc; i++) {\n+\t\tconst char *arg = args->argv[i];\n+\n+\t\t/* process want */\n+\t\tif (parse_want(arg))\n+\t\t\tcontinue;\n+\t\t/* process have line */\n+\t\tif (parse_have(arg, &data->haves))\n+\t\t\tcontinue;\n+\n+\t\t/* process args like thin-pack */\n+\t\tif (!strcmp(arg, \"thin-pack\")) {\n+\t\t\tuse_thin_pack = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"ofs-delta\")) {\n+\t\t\tuse_ofs_delta = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"no-progress\")) {\n+\t\t\tno_progress = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"include-tag\")) {\n+\t\t\tuse_include_tag = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"done\")) {\n+\t\t\tdata->done = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* ignore unknown lines maybe? */\n+\t\tdie(\"unexpect line: '%s'\", arg);\n+\t}\n+}\n+\n+static void read_haves(struct upload_pack_data *data)\n+{\n+\tstruct packet_reader reader;\n+\tpacket_reader_init(&reader, 0, NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n+\n+\t\tif (parse_have(reader.line, &data->haves))\n+\t\t\tcontinue;\n+\t\tif (!strcmp(reader.line, \"done\")) {\n+\t\t\tdata->done = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\tif (reader.status != PACKET_READ_FLUSH)\n+\t\tdie(\"ERROR\");\n+}\n+\n+static int process_haves(struct oid_array *haves, struct oid_array *common)\n+{\n+\tint i;\n+\n+\t/* Process haves */\n+\tfor (i = 0; i < haves->nr; i++) {\n+\t\tconst struct object_id *oid = &haves->oid[i];\n+\t\tstruct object *o;\n+\t\tint we_knew_they_have = 0;\n+\n+\t\tif (!has_object_file(oid))\n+\t\t\tcontinue;\n+\n+\t\toid_array_append(common, oid);\n+\n+\t\to = parse_object(oid);\n+\t\tif (!o)\n+\t\t\tdie(\"oops (%s)\", oid_to_hex(oid));\n+\t\tif (o->type == OBJ_COMMIT) {\n+\t\t\tstruct commit_list *parents;\n+\t\t\tstruct commit *commit = (struct commit *)o;\n+\t\t\tif (o->flags & THEY_HAVE)\n+\t\t\t\twe_knew_they_have = 1;\n+\t\t\telse\n+\t\t\t\to->flags |= THEY_HAVE;\n+\t\t\tif (!oldest_have || (commit->date < oldest_have))\n+\t\t\t\toldest_have = commit->date;\n+\t\t\tfor (parents = commit->parents;\n+\t\t\t parents;\n+\t\t\t parents = parents->next)\n+\t\t\t\tparents->item->object.flags |= THEY_HAVE;\n+\t\t}\n+\t\tif (!we_knew_they_have)\n+\t\t\tadd_object_array(o, NULL, &have_obj);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int send_acks(struct oid_array *acks, struct strbuf *response)\n+{\n+\tint i;\n+\t/* Send Acks */\n+\tif (!acks->nr)\n+\t\tpacket_buf_write(response, \"NAK\\n\");\n+\n+\tfor (i = 0; i < acks->nr; i++) {\n+\t\tpacket_buf_write(response, \"ACK %s common\\n\",\n+\t\t\t\t oid_to_hex(&acks->oid[i]));\n+\t}\n+\n+\tif (ok_to_give_up()) {\n+\t\t/* Send Ready */\n+\t\tpacket_buf_write(response, \"ACK %s ready\\n\",\n+\t\t\t\t oid_to_hex(&acks->oid[i-1]));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_haves_and_send_acks(struct upload_pack_data *data)\n+{\n+\tstruct oid_array common = OID_ARRAY_INIT;\n+\tstruct strbuf response = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tprocess_haves(&data->haves, &common);\n+\tif (data->done) {\n+\t\tret = 1;\n+\t} else if (send_acks(&common, &response)) {\n+\t\tpacket_buf_delim(&response);\n+\t\tret = 1;\n+\t} else {\n+\t\t/* Add Flush */\n+\t\tpacket_buf_flush(&response);\n+\t\tret = 0;\n+\t}\n+\n+\t/* Send response */\n+\twrite_or_die(1, response.buf, response.len);\n+\tstrbuf_release(&response);\n+\n+\toid_array_clear(&data->haves);\n+\toid_array_clear(&common);\n+\treturn ret;\n+}\n+\n+#define FETCH_PROCESS_ARGS 0\n+#define FETCH_READ_HAVES 1\n+#define FETCH_SEND_ACKS 2\n+#define FETCH_SEND_PACK 3\n+#define FETCH_DONE 4\n+\n+int upload_pack_v2(struct repository *r, struct argv_array *keys,\n+\t\t struct argv_array *args)\n+{\n+\tint state = FETCH_PROCESS_ARGS;\n+\tstruct upload_pack_data data = UPLOAD_PACK_DATA_INIT;\n+\tconst char *out;\n+\tuse_sideband = LARGE_PACKET_MAX;\n+\n+\t/* Check if cmd is being run as a stateless-rpc */\n+\tif (has_capability(keys, \"stateless-rpc\", &out))\n+\t\tif (!strcmp(out, \"true\"))\n+\t\t\tdata.stateless_rpc = 1;\n+\n+\twhile (state != FETCH_DONE) {\n+\t\tswitch (state) {\n+\t\tcase FETCH_PROCESS_ARGS:\n+\t\t\tprocess_args(args, &data);\n+\n+\t\t\tif (!want_obj.nr) {\n+\t\t\t\t/*\n+\t\t\t\t * Request didn't contain any 'want' lines,\n+\t\t\t\t * guess they didn't want anything.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\t} else if (data.haves.nr) {\n+\t\t\t\t/*\n+\t\t\t\t * Request had 'have' lines, so lets ACK them.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_SEND_ACKS;\n+\t\t\t} else {\n+\t\t\t\t/*\n+\t\t\t\t * Request had 'want's but no 'have's so we can\n+\t\t\t\t * immedietly go to construct and send a pack.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_SEND_PACK;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase FETCH_READ_HAVES:\n+\t\t\tread_haves(&data);\n+\t\t\tstate = FETCH_SEND_ACKS;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_ACKS:\n+\t\t\tif (process_haves_and_send_acks(&data))\n+\t\t\t\tstate = FETCH_SEND_PACK;\n+\t\t\telse if (data.stateless_rpc)\n+\t\t\t\t/*\n+\t\t\t\t * Request was made via stateless-rpc and a\n+\t\t\t\t * packfile isn't ready to be created and sent.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_READ_HAVES;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_PACK:\n+\t\t\tcreate_pack_file();\n+\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_DONE:\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\tBUG(\"invalid state\");\n+\t\t}\n+\t}\n+\n+\tupload_pack_data_clear(&data);\n+\treturn 0;\n+}\n+\n static int upload_pack_config(const char *var, const char *value, void *unused)\n {\n \tif (!strcmp(\"uploadpack.allowtipsha1inwant\", var)) {\ndiff --git a/upload-pack.h b/upload-pack.h\nnew file mode 100644\nindex 000000000..54c429563\n--- /dev/null\n+++ b/upload-pack.h\n@@ -0,0 +1,9 @@\n+#ifndef UPLOAD_PACK_H\n+#define UPLOAD_PACK_H\n+\n+struct repository;\n+struct argv_array;\n+extern int upload_pack_v2(struct repository *r, struct argv_array *keys,\n+\t\t\t struct argv_array *args);\n+\n+#endif /* UPLOAD_PACK_H */\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335666","messageId":"20180103001828.205012-25-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 24/26] pkt-line: add packet_buf_write_len function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:26Z","receivedAt":"2018-01-03T00:19:55Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add the 'packet_buf_write_len()' function which allows for writing an\narbitrary length buffer into a 'struct strbuf' and formatting it in\npacket-line format.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 16 ++++++++++++++++\n pkt-line.h | 1 +\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 3159cbe10..e9968b7df 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -215,6 +215,22 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n \tva_end(args);\n }\n \n+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len)\n+{\n+\tsize_t orig_len, n;\n+\n+\torig_len = buf->len;\n+\tstrbuf_addstr(buf, \"0000\");\n+\tstrbuf_add(buf, data, len);\n+\tn = buf->len - orig_len;\n+\n+\tif (n > LARGE_PACKET_MAX)\n+\t\tdie(\"protocol error: impossibly long line\");\n+\n+\tset_packet_header(&buf->buf[orig_len], n);\n+\tpacket_trace(buf->buf + orig_len + 4, n - 4, 1);\n+}\n+\n int write_packetized_from_fd(int fd_in, int fd_out)\n {\n \tstatic char buf[LARGE_PACKET_DATA_MAX];\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 97b6dd1c7..d411fcb30 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -26,6 +26,7 @@ void packet_buf_flush(struct strbuf *buf);\n void packet_buf_delim(struct strbuf *buf);\n void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len);\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int write_packetized_from_fd(int fd_in, int fd_out);\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335667","messageId":"20180103001828.205012-16-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 15/26] transport: convert transport_get_remote_refs to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:17Z","receivedAt":"2018-01-03T00:20:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert 'transport_get_remote_refs()' to optionally take a list of ref\npatterns.\n\nSigned-off-by: Brandon Williams \n---\n builtin/clone.c | 2 +-\n builtin/fetch.c | 4 ++--\n builtin/ls-remote.c | 2 +-\n builtin/remote.c | 2 +-\n transport.c | 7 +++++--\n transport.h | 3 ++-\n 6 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 2da71db10..4db3079ac 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1104,7 +1104,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (transport->smart_options && !deepen)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n-\trefs = transport_get_remote_refs(transport);\n+\trefs = transport_get_remote_refs(transport, NULL);\n \n \tif (refs) {\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7bbcd26fa..850382f55 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -250,7 +250,7 @@ static void find_non_local_tags(struct transport *transport,\n \tstruct string_list_item *item = NULL;\n \n \tfor_each_ref(add_existing, &existing_refs);\n-\tfor (ref = transport_get_remote_refs(transport); ref; ref = ref->next) {\n+\tfor (ref = transport_get_remote_refs(transport, NULL); ref; ref = ref->next) {\n \t\tif (!starts_with(ref->name, \"refs/tags/\"))\n \t\t\tcontinue;\n \n@@ -336,7 +336,7 @@ static struct ref *get_ref_map(struct transport *transport,\n \t/* opportunistically-updated references: */\n \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n \n-\tconst struct ref *remote_refs = transport_get_remote_refs(transport);\n+\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n \n \tif (refspec_count) {\n \t\tstruct refspec *fetch_refspec;\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex c4be98ab9..c6e9847c5 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -96,7 +96,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (uploadpack != NULL)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n \n-\tref = transport_get_remote_refs(transport);\n+\tref = transport_get_remote_refs(transport, NULL);\n \tif (transport_disconnect(transport))\n \t\treturn 1;\n \ndiff --git a/builtin/remote.c b/builtin/remote.c\nindex d95bf904c..d0b6ff6e2 100644\n--- a/builtin/remote.c\n+++ b/builtin/remote.c\n@@ -862,7 +862,7 @@ static int get_remote_ref_states(const char *name,\n \tif (query) {\n \t\ttransport = transport_get(states->remote, states->remote->url_nr > 0 ?\n \t\t\tstates->remote->url[0] : NULL);\n-\t\tremote_refs = transport_get_remote_refs(transport);\n+\t\tremote_refs = transport_get_remote_refs(transport, NULL);\n \t\ttransport_disconnect(transport);\n \n \t\tstates->queried = 1;\ndiff --git a/transport.c b/transport.c\nindex c54a44630..dfc603b36 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1136,10 +1136,13 @@ int transport_push(struct transport *transport,\n \treturn 1;\n }\n \n-const struct ref *transport_get_remote_refs(struct transport *transport)\n+const struct ref *transport_get_remote_refs(struct transport *transport,\n+\t\t\t\t\t const struct argv_array *ref_patterns)\n {\n \tif (!transport->got_remote_refs) {\n-\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n+\t\ttransport->remote_refs =\n+\t\t\ttransport->vtable->get_refs_list(transport, 0,\n+\t\t\t\t\t\t\t ref_patterns);\n \t\ttransport->got_remote_refs = 1;\n \t}\n \ndiff --git a/transport.h b/transport.h\nindex 731c78b67..4b656f315 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -178,7 +178,8 @@ int transport_push(struct transport *connection,\n \t\t int refspec_nr, const char **refspec, int flags,\n \t\t unsigned int * reject_reasons);\n \n-const struct ref *transport_get_remote_refs(struct transport *transport);\n+const struct ref *transport_get_remote_refs(struct transport *transport,\n+\t\t\t\t\t const struct argv_array *ref_patterns);\n \n int transport_fetch_refs(struct transport *transport, struct ref *refs);\n void transport_unlock_pack(struct transport *transport);\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335668","messageId":"20180103001828.205012-18-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 17/26] fetch: pass ref patterns when fetching","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:19Z","receivedAt":"2018-01-03T00:20:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct a list of ref patterns to be passed to\n'transport_get_remote_refs()' from the refspec to be used during the\nfetch. This list of ref patterns will be used to allow the server to\nfilter the ref advertisement when communicating using protocol v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch.c | 12 +++++++++++-\n 1 file changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 850382f55..8128450bf 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -332,11 +332,21 @@ static struct ref *get_ref_map(struct transport *transport,\n \tstruct ref *rm;\n \tstruct ref *ref_map = NULL;\n \tstruct ref **tail = &ref_map;\n+\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n \n \t/* opportunistically-updated references: */\n \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n \n-\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n+\tconst struct ref *remote_refs;\n+\n+\tfor (i = 0; i < refspec_count; i++) {\n+\t\tif (!refspecs[i].exact_sha1)\n+\t\t\targv_array_push(&ref_patterns, refspecs[i].src);\n+\t}\n+\n+\tremote_refs = transport_get_remote_refs(transport, &ref_patterns);\n+\n+\targv_array_clear(&ref_patterns);\n \n \tif (refspec_count) {\n \t\tstruct refspec *fetch_refspec;\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335669","messageId":"20180103001828.205012-7-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 06/26] transport: use get_refs_via_connect to get refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:08Z","receivedAt":"2018-01-03T00:20:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Remove code duplication and use the existing 'get_refs_via_connect()'\nfunction to retrieve a remote's heads in 'fetch_refs_via_pack()' and\n'git_transport_push()'.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 18 ++++--------------\n 1 file changed, 4 insertions(+), 14 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex fc802260f..8e8779096 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n \n-\tif (!data->got_remote_heads) {\n-\t\tconnect_setup(transport, 0);\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL, &data->shallow);\n-\t\tdata->got_remote_heads = 1;\n-\t}\n+\tif (!data->got_remote_heads)\n+\t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n \trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n@@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tstruct send_pack_args args;\n \tint ret;\n \n-\tif (!data->got_remote_heads) {\n-\t\tstruct ref *tmp_refs;\n-\t\tconnect_setup(transport, 1);\n-\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n-\t\t\t\t NULL, &data->shallow);\n-\t\tdata->got_remote_heads = 1;\n-\t}\n+\tif (!data->got_remote_heads)\n+\t\tget_refs_via_connect(transport, 1);\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335670","messageId":"20180103001828.205012-12-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 11/26] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:13Z","receivedAt":"2018-01-03T00:20:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce git-serve, the base server for protocol version 2.\n\nProtocol version 2 is intended to be a replacement for Git's current\nwire protocol. The intention is that it will be a simpler, less\nwasteful protocol which can evolve over time.\n\nProtocol version 2 improves upon version 1 by eliminating the initial\nref advertisement. In its place a server will export a list of\ncapabilities and commands which it supports in a capability\nadvertisement. A client can then request that a particular command be\nexecuted by providing a number of capabilities and command specific\nparameters. At the completion of a command, a client can request that\nanother command be executed or can terminate the connection by sending a\nflush packet.\n\nSigned-off-by: Brandon Williams \n---\n .gitignore | 1 +\n Documentation/technical/protocol-v2.txt | 91 ++++++++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/serve.c | 30 ++++\n git.c | 1 +\n serve.c | 239 ++++++++++++++++++++++++++++++++\n serve.h | 15 ++\n 8 files changed, 380 insertions(+)\n create mode 100644 Documentation/technical/protocol-v2.txt\n create mode 100644 builtin/serve.c\n create mode 100644 serve.c\n create mode 100644 serve.h\n\ndiff --git a/.gitignore b/.gitignore\nindex 833ef3b0b..2d0450c26 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -140,6 +140,7 @@\n /git-rm\n /git-send-email\n /git-send-pack\n+/git-serve\n /git-sh-i18n\n /git-sh-i18n--envsubst\n /git-sh-setup\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nnew file mode 100644\nindex 000000000..b87ba3816\n--- /dev/null\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -0,0 +1,91 @@\n+ Git Wire Protocol, Version 2\n+==============================\n+\n+This document presents a specification for a version 2 of Git's wire\n+protocol. Protocol v2 will improve upon v1 in the following ways:\n+\n+ * Instead of multiple service names, multiple commands will be\n+ supported by a single service.\n+ * Easily extendable as capabilities are moved into their own section\n+ of the protocol, no longer being hidden behind a NUL byte and\n+ limited by the size of a pkt-line (as there will be a single\n+ capability per pkt-line).\n+ * Separate out other information hidden behind NUL bytes (e.g. agent\n+ string as a capability and symrefs can be requested using 'ls-refs')\n+ * Reference advertisement will be omitted unless explicitly requested\n+ * ls-refs command to explicitly request some refs\n+\n+ Detailed Design\n+=================\n+\n+A client can request to speak protocol v2 by sending `version=2` in the\n+side-channel `GIT_PROTOCOL` in the initial request to the server.\n+\n+In protocol v2 communication is command oriented. When first contacting a\n+server a list of capabilities will advertised. Some of these capabilities\n+will be commands which a client can request be executed. Once a command\n+has completed, a client can reuse the connection and request that other\n+commands be executed.\n+\n+ Special Packets\n+-----------------\n+\n+In protocol v2 these special packets will have the following semantics:\n+\n+ * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n+ * '0001' Delimiter Packet (delim-pkt) - separates sections of a message\n+\n+ Capability Advertisement\n+--------------------------\n+\n+A server which decides to communicate (based on a request from a client)\n+using protocol version 2, notifies the client by sending a version string\n+in its initial response followed by an advertisement of its capabilities.\n+Each capability is a key with an optional value. Clients must ignore all\n+unknown keys. Semantics of unknown values are left to the definition of\n+each key. Some capabilities will describe commands which can be requested\n+to be executed by the client.\n+\n+ capability-advertisement = protocol-version\n+\t\t\t capability-list\n+\t\t\t flush-pkt\n+\n+ protocol-version = PKT-LINE(\"version 2\" LF)\n+ capability-list = *capability\n+ capability = PKT-LINE(key[=value] LF)\n+\n+ key = 1*CHAR\n+ value = 1*CHAR\n+ CHAR = 1*(ALPHA / DIGIT / \"-\" / \"_\")\n+\n+A client then responds to select the command it wants with any particular\n+capabilities or arguments. There is then an optional section where the\n+client can provide any command specific parameters or queries.\n+\n+ command-request = command\n+\t\t capability-list\n+\t\t (command-args)\n+\t\t flush-pkt\n+ command = PKT-LINE(\"command=\" key LF)\n+ command-args = delim-pkt\n+\t\t *arg\n+ arg = 1*CHAR\n+\n+The server will then check to ensure that the client's request is\n+comprised of a valid command as well as valid capabilities which were\n+advertised. If the request is valid the server will then execute the\n+command.\n+\n+A particular command can last for as many rounds as are required to\n+complete the service (multiple for negotiation during fetch or no\n+additional trips in the case of ls-refs).\n+\n+When finished a client should send an empty request of just a flush-pkt to\n+terminate the connection.\n+\n+ Commands in v2\n+~~~~~~~~~~~~~~~~\n+\n+Commands are the core actions that a client wants to perform (fetch, push,\n+etc). Each command will be provided with a list capabilities and\n+arguments as requested by a client.\ndiff --git a/Makefile b/Makefile\nindex e0740b452..5f3b5fe8b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -876,6 +876,7 @@ LIB_OBJS += revision.o\n LIB_OBJS += run-command.o\n LIB_OBJS += send-pack.o\n LIB_OBJS += sequencer.o\n+LIB_OBJS += serve.o\n LIB_OBJS += server-info.o\n LIB_OBJS += setup.o\n LIB_OBJS += sha1-array.o\n@@ -1009,6 +1010,7 @@ BUILTIN_OBJS += builtin/rev-parse.o\n BUILTIN_OBJS += builtin/revert.o\n BUILTIN_OBJS += builtin/rm.o\n BUILTIN_OBJS += builtin/send-pack.o\n+BUILTIN_OBJS += builtin/serve.o\n BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-ref.o\ndiff --git a/builtin.h b/builtin.h\nindex f332a1257..3f3fdfc28 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -215,6 +215,7 @@ extern int cmd_rev_parse(int argc, const char **argv, const char *prefix);\n extern int cmd_revert(int argc, const char **argv, const char *prefix);\n extern int cmd_rm(int argc, const char **argv, const char *prefix);\n extern int cmd_send_pack(int argc, const char **argv, const char *prefix);\n+extern int cmd_serve(int argc, const char **argv, const char *prefix);\n extern int cmd_shortlog(int argc, const char **argv, const char *prefix);\n extern int cmd_show(int argc, const char **argv, const char *prefix);\n extern int cmd_show_branch(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/serve.c b/builtin/serve.c\nnew file mode 100644\nindex 000000000..bb726786a\n--- /dev/null\n+++ b/builtin/serve.c\n@@ -0,0 +1,30 @@\n+#include \"cache.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"serve.h\"\n+\n+static char const * const grep_usage[] = {\n+\tN_(\"git serve []\"),\n+\tNULL\n+};\n+\n+int cmd_serve(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct serve_options opts = SERVE_OPTIONS_INIT;\n+\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n+\t\t\t N_(\"quit after a single request/response exchange\")),\n+\t\tOPT_BOOL(0, \"advertise-capabilities\", &opts.advertise_capabilities,\n+\t\t\t N_(\"exit immediately after advertising capabilities\")),\n+\t\tOPT_END()\n+\t};\n+\n+\t/* ignore all unknown cmdline switches for now */\n+\targc = parse_options(argc, argv, prefix, options, grep_usage,\n+\t\t\t PARSE_OPT_KEEP_DASHDASH |\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tserve(&opts);\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex f71073dc8..f85d682b6 100644\n--- a/git.c\n+++ b/git.c\n@@ -461,6 +461,7 @@ static struct cmd_struct commands[] = {\n \t{ \"revert\", cmd_revert, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"rm\", cmd_rm, RUN_SETUP },\n \t{ \"send-pack\", cmd_send_pack, RUN_SETUP },\n+\t{ \"serve\", cmd_serve, RUN_SETUP },\n \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n \t{ \"show\", cmd_show, RUN_SETUP },\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\ndiff --git a/serve.c b/serve.c\nnew file mode 100644\nindex 000000000..da8127775\n--- /dev/null\n+++ b/serve.c\n@@ -0,0 +1,239 @@\n+#include \"cache.h\"\n+#include \"repository.h\"\n+#include \"config.h\"\n+#include \"pkt-line.h\"\n+#include \"version.h\"\n+#include \"argv-array.h\"\n+#include \"serve.h\"\n+\n+static int always_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\treturn 1;\n+}\n+\n+static int agent_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, git_user_agent_sanitized());\n+\treturn 1;\n+}\n+\n+struct protocol_capability {\n+\tconst char *name; /* capability name */\n+\n+\t/*\n+\t * Function queried to see if a capability should be advertised.\n+\t * Optionally a value can be specified by adding it to 'value'.\n+\t */\n+\tint (*advertise)(struct repository *r, struct strbuf *value);\n+\n+\t/*\n+\t * Function called when a client requests the capability as a command.\n+\t * The command request will be provided to the function via 'keys', the\n+\t * capabilities requested, and 'args', the command specific parameters.\n+\t *\n+\t * This field should be NULL for capabilities which are not commands.\n+\t */\n+\tint (*command)(struct repository *r,\n+\t\t struct argv_array *keys,\n+\t\t struct argv_array *args);\n+};\n+\n+static struct protocol_capability capabilities[] = {\n+\t{ \"agent\", agent_advertise, NULL },\n+\t{ \"stateless-rpc\", always_advertise, NULL },\n+};\n+\n+static void advertise_capabilities(void)\n+{\n+\tstruct strbuf capability = STRBUF_INIT;\n+\tstruct strbuf value = STRBUF_INIT;\n+\tint i;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n+\t\tstruct protocol_capability *c = &capabilities[i];\n+\n+\t\tif (c->advertise(the_repository, &value)) {\n+\t\t\tstrbuf_addstr(&capability, c->name);\n+\n+\t\t\tif (value.len) {\n+\t\t\t\tstrbuf_addch(&capability, '=');\n+\t\t\t\tstrbuf_addbuf(&capability, &value);\n+\t\t\t}\n+\n+\t\t\tstrbuf_addch(&capability, '\\n');\n+\t\t\tpacket_write(1, capability.buf, capability.len);\n+\t\t}\n+\n+\t\tstrbuf_reset(&capability);\n+\t\tstrbuf_reset(&value);\n+\t}\n+\n+\tpacket_flush(1);\n+\tstrbuf_release(&capability);\n+\tstrbuf_release(&value);\n+}\n+\n+static struct protocol_capability *get_capability(const char *key)\n+{\n+\tint i;\n+\n+\tif (!key)\n+\t\treturn NULL;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n+\t\tstruct protocol_capability *c = &capabilities[i];\n+\t\tconst char *out;\n+\t\tif (skip_prefix(key, c->name, &out) && (!*out || *out == '='))\n+\t\t\treturn c;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static int is_valid_capability(const char *key)\n+{\n+\tconst struct protocol_capability *c = get_capability(key);\n+\n+\treturn c && c->advertise(the_repository, NULL);\n+}\n+\n+static int is_command(const char *key, struct protocol_capability **command)\n+{\n+\tconst char *out;\n+\n+\tif (skip_prefix(key, \"command=\", &out)) {\n+\t\tstruct protocol_capability *cmd = get_capability(out);\n+\n+\t\tif (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)\n+\t\t\tdie(\"invalid cmd '%s'\", out);\n+\t\tif (*command)\n+\t\t\tdie(\"command already requested\");\n+\n+\t\t*command = cmd;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int has_capability(const struct argv_array *keys, const char *capability,\n+\t\t const char **value)\n+{\n+\tint i;\n+\tfor (i = 0; i < keys->argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(keys->argv[i], capability, &out) &&\n+\t\t (!*out || *out == '=')) {\n+\t\t\tif (value) {\n+\t\t\t\tif (*out == '=')\n+\t\t\t\t\tout++;\n+\t\t\t\t*value = out;\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+#define PROCESS_REQUEST_KEYS 0\n+#define PROCESS_REQUEST_ARGS 1\n+#define PROCESS_REQUEST_DONE 2\n+\n+static int process_request(void)\n+{\n+\tint state = PROCESS_REQUEST_KEYS;\n+\tstruct packet_reader reader;\n+\tstruct argv_array keys = ARGV_ARRAY_INIT;\n+\tstruct argv_array args = ARGV_ARRAY_INIT;\n+\tstruct protocol_capability *command = NULL;\n+\n+\tpacket_reader_init(&reader, 0, NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (state != PROCESS_REQUEST_DONE) {\n+\t\tswitch (packet_reader_read(&reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tBUG(\"Should have already died when seeing EOF\");\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tstate = PROCESS_REQUEST_DONE;\n+\t\t\tcontinue;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tif (state != PROCESS_REQUEST_KEYS)\n+\t\t\t\tdie(\"protocol error\");\n+\t\t\tstate = PROCESS_REQUEST_ARGS;\n+\t\t\t/*\n+\t\t\t * maybe include a check to make sure that a\n+\t\t\t * command/capabilities were given.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (state) {\n+\t\tcase PROCESS_REQUEST_KEYS:\n+\t\t\t/* collect request; a sequence of keys and values */\n+\t\t\tif (is_command(reader.line, &command) ||\n+\t\t\t is_valid_capability(reader.line))\n+\t\t\t\targv_array_push(&keys, reader.line);\n+\t\t\tbreak;\n+\t\tcase PROCESS_REQUEST_ARGS:\n+\t\t\t/* collect arguments for the requested command */\n+\t\t\targv_array_push(&args, reader.line);\n+\t\t\tbreak;\n+\t\tcase PROCESS_REQUEST_DONE:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tBUG(\"invalid state\");\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If no command and no keys were given then the client wanted to\n+\t * terminate the connection.\n+\t */\n+\tif (!keys.argc && !args.argc)\n+\t\treturn 1;\n+\n+\tif (!command)\n+\t\tdie(\"no command requested\");\n+\n+\tcommand->command(the_repository, &keys, &args);\n+\n+\targv_array_clear(&keys);\n+\targv_array_clear(&args);\n+\treturn 0;\n+}\n+\n+/* Main serve loop for protocol version 2 */\n+void serve(struct serve_options *options)\n+{\n+\tif (options->advertise_capabilities || !options->stateless_rpc) {\n+\t\t/* serve by default supports v2 */\n+\t\tpacket_write_fmt(1, \"version 2\\n\");\n+\n+\t\tadvertise_capabilities();\n+\t\t/*\n+\t\t * If only the list of capabilities was requested exit\n+\t\t * immediately after advertising capabilities\n+\t\t */\n+\t\tif (options->advertise_capabilities)\n+\t\t\treturn;\n+\t}\n+\n+\t/*\n+\t * If stateless-rpc was requested then exit after\n+\t * a single request/response exchange\n+\t */\n+\tif (options->stateless_rpc) {\n+\t\tprocess_request();\n+\t} else {\n+\t\tfor (;;)\n+\t\t\tif (process_request())\n+\t\t\t\tbreak;\n+\t}\n+}\ndiff --git a/serve.h b/serve.h\nnew file mode 100644\nindex 000000000..fe65ba9f4\n--- /dev/null\n+++ b/serve.h\n@@ -0,0 +1,15 @@\n+#ifndef SERVE_H\n+#define SERVE_H\n+\n+struct argv_array;\n+extern int has_capability(const struct argv_array *keys, const char *capability,\n+\t\t\t const char **value);\n+\n+struct serve_options {\n+\tunsigned advertise_capabilities;\n+\tunsigned stateless_rpc;\n+};\n+#define SERVE_OPTIONS_INIT { 0 }\n+extern void serve(struct serve_options *options);\n+\n+#endif /* SERVE_H */\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335671","messageId":"20180103001828.205012-10-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 09/26] transport: store protocol version","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:11Z","receivedAt":"2018-01-03T00:20:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Once protocol_v2 is introduced requesting a fetch or a push will need to\nbe handled differently depending on the protocol version. Store the\nprotocol version the server is speaking in 'struct git_transport_data'\nand use it to determine what to do in the case of a fetch or a push.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 35 ++++++++++++++++++++++++++---------\n 1 file changed, 26 insertions(+), 9 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex 63c3dbab9..2378dcb38 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -118,6 +118,7 @@ struct git_transport_data {\n \tstruct child_process *conn;\n \tint fd[2];\n \tunsigned got_remote_heads : 1;\n+\tenum protocol_version version;\n \tstruct oid_array extra_have;\n \tstruct oid_array shallow;\n };\n@@ -200,7 +201,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n-\tswitch (discover_version(&reader)) {\n+\tdata->version = discover_version(&reader);\n+\tswitch (data->version) {\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &refs,\n@@ -221,7 +223,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n {\n \tint ret = 0;\n \tstruct git_transport_data *data = transport->data;\n-\tstruct ref *refs;\n+\tstruct ref *refs = NULL;\n \tchar *dest = xstrdup(transport->url);\n \tstruct fetch_pack_args args;\n \tstruct ref *refs_tmp = NULL;\n@@ -247,10 +249,18 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tif (!data->got_remote_heads)\n \t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n-\trefs = fetch_pack(&args, data->fd, data->conn,\n-\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n-\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n-\t\t\t &transport->pack_lockfile);\n+\tswitch (data->version) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\trefs = fetch_pack(&args, data->fd, data->conn,\n+\t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n+\t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n+\t\t\t\t &transport->pack_lockfile);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \tclose(data->fd[0]);\n \tclose(data->fd[1]);\n \tif (finish_connect(data->conn))\n@@ -549,7 +559,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n {\n \tstruct git_transport_data *data = transport->data;\n \tstruct send_pack_args args;\n-\tint ret;\n+\tint ret = 0;\n \n \tif (!data->got_remote_heads)\n \t\tget_refs_via_connect(transport, 1);\n@@ -574,8 +584,15 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \telse\n \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n \n-\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n-\t\t\t&data->extra_have);\n+\tswitch (data->version) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n+\t\t\t\t&data->extra_have);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \tclose(data->fd[1]);\n \tclose(data->fd[0]);\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335672","messageId":"20180103001828.205012-11-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH 10/26] protocol: introduce enum protocol_version value protocol_v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T00:18:12Z","receivedAt":"2018-01-03T00:20:19Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce protocol_v2, a new value for 'enum protocol_version'.\nSubsequent patches will fill in the implementation of protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 3 +++\n builtin/receive-pack.c | 6 ++++++\n builtin/send-pack.c | 3 +++\n connect.c | 3 +++\n protocol.c | 2 ++\n protocol.h | 1 +\n remote-curl.c | 3 +++\n transport.c | 9 +++++++++\n upload-pack.c | 6 ++++++\n 9 files changed, 36 insertions(+)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 85d4faf76..f492e8abd 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex b7ce7c7f5..3656e94fd 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tunpack_limit = receive_unpack_limit;\n \n \tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v2:\n+\t\t/*\n+\t\t * push support for protocol v2 has not been implemented yet,\n+\t\t * so ignore the request to use v2 and fallback to using v0.\n+\t\t */\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 83cb125a6..b5427f75e 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\ndiff --git a/connect.c b/connect.c\nindex 1787b0212..caa539b75 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)\n \n \t/* Maybe process capabilities here, at least for v2 */\n \tswitch (version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/* Read the peeked version line */\n \t\tpacket_reader_read(reader);\ndiff --git a/protocol.c b/protocol.c\nindex 43012b7eb..5e636785d 100644\n--- a/protocol.c\n+++ b/protocol.c\n@@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)\n \t\treturn protocol_v0;\n \telse if (!strcmp(value, \"1\"))\n \t\treturn protocol_v1;\n+\telse if (!strcmp(value, \"2\"))\n+\t\treturn protocol_v2;\n \telse\n \t\treturn protocol_unknown_version;\n }\ndiff --git a/protocol.h b/protocol.h\nindex 1b2bc94a8..2ad35e433 100644\n--- a/protocol.h\n+++ b/protocol.h\n@@ -5,6 +5,7 @@ enum protocol_version {\n \tprotocol_unknown_version = -1,\n \tprotocol_v0 = 0,\n \tprotocol_v1 = 1,\n+\tprotocol_v2 = 2,\n };\n \n /*\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 9f6d07683..dae8a4a48 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\ndiff --git a/transport.c b/transport.c\nindex 2378dcb38..83d9dd1df 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &refs,\n@@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\trefs = fetch_pack(&args, data->fd, data->conn,\n@@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n \n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 9a507ae53..2bc888fc1 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1104,6 +1104,12 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \tgit_config(upload_pack_config, NULL);\n \n \tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v2:\n+\t\t/*\n+\t\t * fetch support for protocol v2 has not been implemented yet,\n+\t\t * so ignore the request to use v2 and fallback to using v0.\n+\t\t */\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\n-- \n2.15.1.620.gb9897f4670-goog\n\n"},{"id":"335771","messageId":"CAGZ79kbd1_nnYEJVN8a_NuwrEfHbT0wiwEfdkUz-kZRdpstqHA@mail.gmail.com","threadId":"47524","inReplyTo":"20180103001828.205012-2-bmwill@google.com","subject":"Re: [PATCH 01/26] pkt-line: introduce packet_read_with_status","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-03T19:27:47Z","receivedAt":"2018-01-03T19:27:53Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> The current pkt-line API encodes the status of a pkt-line read in the\n> length of the read content. An error is indicated with '-1', a flush\n> with '0' (which can be confusing since a return value of '0' can also\n> indicate an empty pkt-line), and a positive integer for the length of\n> the read content otherwise. This doesn't leave much room for allowing\n> the addition of additional special packets in the future.\n>\n> To solve this introduce 'packet_read_with_status()' which reads a packet\n> and returns the status of the read encoded as an 'enum packet_status'\n> type. This allows for easily identifying between special and normal\n> packets as well as errors. It also enables easily adding a new special\n> packet in the future.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> pkt-line.c | 55 ++++++++++++++++++++++++++++++++++++++++++-------------\n> pkt-line.h | 15 +++++++++++++++\n> 2 files changed, 57 insertions(+), 13 deletions(-)\n>\n> diff --git a/pkt-line.c b/pkt-line.c\n> index 2827ca772..8d7cd389f 100644\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -280,28 +280,33 @@ static int packet_length(const char *linelen)\n> return (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);\n> }\n>\n> -int packet_read(int fd, char **src_buf, size_t *src_len,\n> - char *buffer, unsigned size, int options)\n> +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n> + char *buffer, unsigned size, int *pktlen,\n> + int options)\n> {\n> - int len, ret;\n> + int len;\n> char linelen[4];\n>\n> - ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);\n> - if (ret < 0)\n> - return ret;\n> + if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)\n> + return PACKET_READ_EOF;\n> +\n> len = packet_length(linelen);\n> if (len < 0)\n> die(\"protocol error: bad line length character: %.4s\", linelen);\n> - if (!len) {\n> +\n> + if (len == 0) {\n> packet_trace(\"0000\", 4, 0);\n> - return 0;\n> + return PACKET_READ_FLUSH;\n> + } else if (len >= 1 && len <= 3) {\n> + die(\"protocol error: bad line length character: %.4s\", linelen);\n\nI wonder how much libified code we want here already, maybe we could\nhave PACKET_READ_ERROR as a return value here instead of die()ing.\nThere could also be an option that tells this code to die on error, this reminds\nme of the repository discovery as well as the refs code, both of which have\nthis pattern.\n\nCurrently this series is only upgrading commands that use the network\nanyway, so I guess die()ing in an ls-remote or fetch is no big deal,\nbut it could\nbe interesting to keep going once we have more of the partial clone\nstuff working\n(e.g. remote assisted log/blame would want to gracefully fall back instead of\ndie()ing without any useful output, I would think.)\n\n> }\n> +\n> len -= 4;\n> - if (len >= size)\n> + if ((len < 0) || ((unsigned)len >= size))\n> die(\"protocol error: bad line length %d\", len);\n> - ret = get_packet_data(fd, src_buf, src_len, buffer, len, options);\n> - if (ret < 0)\n> - return ret;\n> +\n> + if (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)\n> + return PACKET_READ_EOF;\n>\n> if ((options & PACKET_READ_CHOMP_NEWLINE) &&\n> len && buffer[len-1] == '\\n')\n> @@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,\n>\n> buffer[len] = 0;\n> packet_trace(buffer, len, 0);\n> - return len;\n> + *pktlen = len;\n> + return PACKET_READ_NORMAL;\n> +}\n> +\n> +int packet_read(int fd, char **src_buffer, size_t *src_len,\n> + char *buffer, unsigned size, int options)\n> +{\n> + enum packet_read_status status;\n> + int pktlen;\n> +\n> + status = packet_read_with_status(fd, src_buffer, src_len,\n> + buffer, size, &pktlen,\n> + options);\n> + switch (status) {\n> + case PACKET_READ_EOF:\n> + pktlen = -1;\n> + break;\n> + case PACKET_READ_NORMAL:\n> + break;\n> + case PACKET_READ_FLUSH:\n> + pktlen = 0;\n> + break;\n> + }\n> +\n> + return pktlen;\n> }\n>\n> static char *packet_read_line_generic(int fd,\n> diff --git a/pkt-line.h b/pkt-line.h\n> index 3dad583e2..06c468927 100644\n> --- a/pkt-line.h\n> +++ b/pkt-line.h\n> @@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n> int packet_read(int fd, char **src_buffer, size_t *src_len, char\n> *buffer, unsigned size, int options);\n>\n> +/*\n> + * Read a packetized line into a buffer like the 'packet_read()' function but\n> + * returns an 'enum packet_read_status' which indicates the status of the read.\n> + * The number of bytes read will be assigined to *pktlen if the status of the\n> + * read was 'PACKET_READ_NORMAL'.\n> + */\n> +enum packet_read_status {\n> + PACKET_READ_EOF = -1,\n> + PACKET_READ_NORMAL,\n> + PACKET_READ_FLUSH,\n> +};\n> +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n> + char *buffer, unsigned size, int *pktlen,\n> + int options);\n> +\n> /*\n> * Convenience wrapper for packet_read that is not gentle, and sets the\n> * CHOMP_NEWLINE option. The return value is NULL for a flush packet,\n> --\n> 2.15.1.620.gb9897f4670-goog\n>\n"},{"id":"335772","messageId":"CAGZ79kb2=uU0_K8wr27gNdNX-T+P+7gVdgc5EBdYc3zBobsR8w@mail.gmail.com","threadId":"47524","inReplyTo":"20180103001828.205012-5-bmwill@google.com","subject":"Re: [PATCH 04/26] upload-pack: convert to a builtin","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-03T20:33:37Z","receivedAt":"2018-01-03T20:33:48Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> In order to allow for code sharing with the server-side of fetch in\n> protocol-v2 convert upload-pack to be a builtin.\n\nWhat is the security aspect of this patch?\n\nBy making upload-pack builtin, it gains additional abilities,\nsuch as answers to '-h' or '--help' (which would start a pager).\nIs there an easy way to sooth my concerns? (best put into the\ncommit message)\n\nThanks,\nStefan\n\n>\n> Signed-off-by: Brandon Williams \n> ---\n> Makefile | 3 ++-\n> builtin.h | 1 +\n> git.c | 1 +\n> upload-pack.c | 2 +-\n> 4 files changed, 5 insertions(+), 2 deletions(-)\n>\n> diff --git a/Makefile b/Makefile\n> index 2a81ae22e..e0740b452 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -636,7 +636,6 @@ PROGRAM_OBJS += imap-send.o\n> PROGRAM_OBJS += sh-i18n--envsubst.o\n> PROGRAM_OBJS += shell.o\n> PROGRAM_OBJS += show-index.o\n> -PROGRAM_OBJS += upload-pack.o\n> PROGRAM_OBJS += remote-testsvn.o\n>\n> # Binary suffix, set to .exe for Windows builds\n> @@ -701,6 +700,7 @@ BUILT_INS += git-merge-subtree$X\n> BUILT_INS += git-show$X\n> BUILT_INS += git-stage$X\n> BUILT_INS += git-status$X\n> +BUILT_INS += git-upload-pack$X\n> BUILT_INS += git-whatchanged$X\n>\n> # what 'all' will build and 'install' will install in gitexecdir,\n> @@ -904,6 +904,7 @@ LIB_OBJS += tree-diff.o\n> LIB_OBJS += tree.o\n> LIB_OBJS += tree-walk.o\n> LIB_OBJS += unpack-trees.o\n> +LIB_OBJS += upload-pack.o\n> LIB_OBJS += url.o\n> LIB_OBJS += urlmatch.o\n> LIB_OBJS += usage.o\n> diff --git a/builtin.h b/builtin.h\n> index 42378f3aa..f332a1257 100644\n> --- a/builtin.h\n> +++ b/builtin.h\n> @@ -231,6 +231,7 @@ extern int cmd_update_ref(int argc, const char **argv, const char *prefix);\n> extern int cmd_update_server_info(int argc, const char **argv, const char *prefix);\n> extern int cmd_upload_archive(int argc, const char **argv, const char *prefix);\n> extern int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix);\n> +extern int cmd_upload_pack(int argc, const char **argv, const char *prefix);\n> extern int cmd_var(int argc, const char **argv, const char *prefix);\n> extern int cmd_verify_commit(int argc, const char **argv, const char *prefix);\n> extern int cmd_verify_tag(int argc, const char **argv, const char *prefix);\n> diff --git a/git.c b/git.c\n> index c870b9719..f71073dc8 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -478,6 +478,7 @@ static struct cmd_struct commands[] = {\n> { \"update-server-info\", cmd_update_server_info, RUN_SETUP },\n> { \"upload-archive\", cmd_upload_archive },\n> { \"upload-archive--writer\", cmd_upload_archive_writer },\n> + { \"upload-pack\", cmd_upload_pack },\n> { \"var\", cmd_var, RUN_SETUP_GENTLY },\n> { \"verify-commit\", cmd_verify_commit, RUN_SETUP },\n> { \"verify-pack\", cmd_verify_pack },\n> diff --git a/upload-pack.c b/upload-pack.c\n> index d5de18127..20acaa49d 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -1032,7 +1032,7 @@ static int upload_pack_config(const char *var, const char *value, void *unused)\n> return parse_hide_refs_config(var, value, \"uploadpack\");\n> }\n>\n> -int cmd_main(int argc, const char **argv)\n> +int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n> {\n> const char *dir;\n> int strict = 0;\n> --\n> 2.15.1.620.gb9897f4670-goog\n>\n"},{"id":"335773","messageId":"CAGZ79kaydGBVO=NQuqE-J_pkXGMLN2D42VAuNsPvkkQcK71BLA@mail.gmail.com","threadId":"47524","inReplyTo":"20180103001828.205012-6-bmwill@google.com","subject":"Re: [PATCH 05/26] upload-pack: factor out processing lines","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-03T20:38:23Z","receivedAt":"2018-01-03T20:38:30Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> Factor out the logic for processing shallow, deepen, deepen_since, and\n> deepen_not lines into their own functions to simplify the\n> 'receive_needs()' function in addition to making it easier to reuse some\n> of this logic when implementing protocol_v2.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------\n> 1 file changed, 74 insertions(+), 39 deletions(-)\n>\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 20acaa49d..9a507ae53 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -731,6 +731,75 @@ static void deepen_by_rev_list(int ac, const char **av,\n> packet_flush(1);\n> }\n>\n> +static int process_shallow(const char *line, struct object_array *shallows)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"shallow \", &arg)) {\n\nstylistic nit:\n\n You could invert the condition in each of the process_* functions\n to just have\n\n if (!skip_prefix...))\n return 0\n\n /* less indented code goes here */\n\n return 1;\n\n That way we have less indentation as well as easier code.\n (The reader doesn't need to keep in mind what the else\n part is about; it is a rather local decision to bail out instead\n of having the return at the end of the function.)\n\n\n> + struct object_id oid;\n> + struct object *object;\n> + if (get_oid_hex(arg, &oid))\n> + die(\"invalid shallow line: %s\", line);\n> + object = parse_object(&oid);\n> + if (!object)\n> + return 1;\n> + if (object->type != OBJ_COMMIT)\n> + die(\"invalid shallow object %s\", oid_to_hex(&oid));\n> + if (!(object->flags & CLIENT_SHALLOW)) {\n> + object->flags |= CLIENT_SHALLOW;\n> + add_object_array(object, NULL, shallows);\n> + }\n> + return 1;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +static int process_deepen(const char *line, int *depth)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"deepen \", &arg)) {\n> + char *end = NULL;\n> + *depth = strtol(arg, &end, 0);\n> + if (!end || *end || depth <= 0)\n> + die(\"Invalid deepen: %s\", line);\n> + return 1;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"deepen-since \", &arg)) {\n> + char *end = NULL;\n> + *deepen_since = parse_timestamp(arg, &end, 0);\n> + if (!end || *end || !deepen_since ||\n> + /* revisions.c's max_age -1 is special */\n> + *deepen_since == -1)\n> + die(\"Invalid deepen-since: %s\", line);\n> + *deepen_rev_list = 1;\n> + return 1;\n> + }\n> + return 0;\n> +}\n> +\n> +static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"deepen-not \", &arg)) {\n> + char *ref = NULL;\n> + struct object_id oid;\n> + if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n> + die(\"git upload-pack: ambiguous deepen-not: %s\", line);\n> + string_list_append(deepen_not, ref);\n> + free(ref);\n> + *deepen_rev_list = 1;\n> + return 1;\n> + }\n> + return 0;\n> +}\n> +\n> static void receive_needs(void)\n> {\n> struct object_array shallows = OBJECT_ARRAY_INIT;\n> @@ -752,49 +821,15 @@ static void receive_needs(void)\n> if (!line)\n> break;\n>\n> - if (skip_prefix(line, \"shallow \", &arg)) {\n> - struct object_id oid;\n> - struct object *object;\n> - if (get_oid_hex(arg, &oid))\n> - die(\"invalid shallow line: %s\", line);\n> - object = parse_object(&oid);\n> - if (!object)\n> - continue;\n> - if (object->type != OBJ_COMMIT)\n> - die(\"invalid shallow object %s\", oid_to_hex(&oid));\n> - if (!(object->flags & CLIENT_SHALLOW)) {\n> - object->flags |= CLIENT_SHALLOW;\n> - add_object_array(object, NULL, &shallows);\n> - }\n> + if (process_shallow(line, &shallows))\n> continue;\n> - }\n> - if (skip_prefix(line, \"deepen \", &arg)) {\n> - char *end = NULL;\n> - depth = strtol(arg, &end, 0);\n> - if (!end || *end || depth <= 0)\n> - die(\"Invalid deepen: %s\", line);\n> + if (process_deepen(line, &depth))\n> continue;\n> - }\n> - if (skip_prefix(line, \"deepen-since \", &arg)) {\n> - char *end = NULL;\n> - deepen_since = parse_timestamp(arg, &end, 0);\n> - if (!end || *end || !deepen_since ||\n> - /* revisions.c's max_age -1 is special */\n> - deepen_since == -1)\n> - die(\"Invalid deepen-since: %s\", line);\n> - deepen_rev_list = 1;\n> + if (process_deepen_since(line, &deepen_since, &deepen_rev_list))\n> continue;\n> - }\n> - if (skip_prefix(line, \"deepen-not \", &arg)) {\n> - char *ref = NULL;\n> - struct object_id oid;\n> - if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n> - die(\"git upload-pack: ambiguous deepen-not: %s\", line);\n> - string_list_append(&deepen_not, ref);\n> - free(ref);\n> - deepen_rev_list = 1;\n> + if (process_deepen_not(line, &deepen_not, &deepen_rev_list))\n> continue;\n> - }\n> +\n> if (!skip_prefix(line, \"want \", &arg) ||\n> get_oid_hex(arg, &oid_buf))\n> die(\"git upload-pack: protocol error, \"\n> --\n> 2.15.1.620.gb9897f4670-goog\n>\n"},{"id":"335774","messageId":"20180103203956.GA44365@google.com","threadId":"47524","inReplyTo":"CAGZ79kb2=uU0_K8wr27gNdNX-T+P+7gVdgc5EBdYc3zBobsR8w@mail.gmail.com","subject":"Re: [PATCH 04/26] upload-pack: convert to a builtin","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-03T20:39:56Z","receivedAt":"2018-01-03T20:40:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/03, Stefan Beller wrote:\n> On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> > In order to allow for code sharing with the server-side of fetch in\n> > protocol-v2 convert upload-pack to be a builtin.\n> \n> What is the security aspect of this patch?\n> \n> By making upload-pack builtin, it gains additional abilities,\n> such as answers to '-h' or '--help' (which would start a pager).\n> Is there an easy way to sooth my concerns? (best put into the\n> commit message)\n\nreceive-pack is already a builtin, so theres that.\n\n> \n> Thanks,\n> Stefan\n> \n\n-- \nBrandon Williams\n"},{"id":"335787","messageId":"CAGZ79kbLJp=FDssowayYzyyHmRK65hDXeznvPbHLx5f6i0u3_w@mail.gmail.com","threadId":"47524","inReplyTo":"20180103001828.205012-7-bmwill@google.com","subject":"Re: [PATCH 06/26] transport: use get_refs_via_connect to get refs","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-03T21:20:20Z","receivedAt":"2018-01-03T21:20:26Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> Remove code duplication and use the existing 'get_refs_via_connect()'\n> function to retrieve a remote's heads in 'fetch_refs_via_pack()' and\n> 'git_transport_push()'.\n>\n> Signed-off-by: Brandon Williams \n\nReviewed-by: Stefan Beller \n\n> ---\n> transport.c | 18 ++++--------------\n> 1 file changed, 4 insertions(+), 14 deletions(-)\n>\n> diff --git a/transport.c b/transport.c\n> index fc802260f..8e8779096 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n> args.cloning = transport->cloning;\n> args.update_shallow = data->options.update_shallow;\n>\n> - if (!data->got_remote_heads) {\n> - connect_setup(transport, 0);\n> - get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n> - NULL, &data->shallow);\n> - data->got_remote_heads = 1;\n> - }\n> + if (!data->got_remote_heads)\n> + refs_tmp = get_refs_via_connect(transport, 0);\n>\n> refs = fetch_pack(&args, data->fd, data->conn,\n> refs_tmp ? refs_tmp : transport->remote_refs,\n> @@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n> struct send_pack_args args;\n> int ret;\n>\n> - if (!data->got_remote_heads) {\n> - struct ref *tmp_refs;\n> - connect_setup(transport, 1);\n> -\n> - get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n> - NULL, &data->shallow);\n> - data->got_remote_heads = 1;\n> - }\n> + if (!data->got_remote_heads)\n> + get_refs_via_connect(transport, 1);\n>\n> memset(&args, 0, sizeof(args));\n> args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n> --\n> 2.15.1.620.gb9897f4670-goog\n>\n"},{"id":"335807","messageId":"CAGZ79kaQMBaA+gxqjY0xBNZiy2PdafoXjB+kowM9pj9heLwRPA@mail.gmail.com","threadId":"47524","inReplyTo":"20180103001828.205012-13-bmwill@google.com","subject":"Re: [PATCH 12/26] ls-refs: introduce ls-refs server command","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-04T00:17:11Z","receivedAt":"2018-01-04T00:17:18Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> Introduce the ls-refs server command. In protocol v2, the ls-refs\n> command is used to request the ref advertisement from the server. Since\n> it is a command which can be requested (as opposed to mandatory in v1),\n> a client can sent a number of parameters in its request to limit the ref\n> advertisement based on provided ref-patterns.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> Documentation/technical/protocol-v2.txt | 26 +++++++++\n> Makefile | 1 +\n> ls-refs.c | 97 +++++++++++++++++++++++++++++++++\n> ls-refs.h | 9 +++\n\nMaybe consider putting any served command into a sub directory?\n\nFor example the code in builtin/ has laxer rules w.r.t. die()ing\nas it is a user facing command, whereas some devs want to see\ncode at the root of the repo to not die() at all as the eventual goal\nis to have a library there.\nAll this code is on the remote side, which also has different traits than\nthe code at the root of the git.git repo; non-localisation comes to mind,\nbut there might be other aspects as well (security?).\n\n\n> serve.c | 2 +\n> 5 files changed, 135 insertions(+)\n> create mode 100644 ls-refs.c\n> create mode 100644 ls-refs.h\n>\n> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> index b87ba3816..5f4d0e719 100644\n> --- a/Documentation/technical/protocol-v2.txt\n> +++ b/Documentation/technical/protocol-v2.txt\n> @@ -89,3 +89,29 @@ terminate the connection.\n> Commands are the core actions that a client wants to perform (fetch, push,\n> etc). Each command will be provided with a list capabilities and\n> arguments as requested by a client.\n> +\n> + Ls-refs\n\nSo is it ls-refs or Ls-refs or is any capitalization valid?\n\n> +---------\n> +\n> +Ls-refs is the command used to request a reference advertisement in v2.\n> +Unlike the current reference advertisement, ls-refs takes in parameters\n> +which can be used to limit the refs sent from the server.\n> +\n> +Ls-ref takes in the following parameters wraped in packet-lines:\n> +\n> + symrefs: In addition to the object pointed by it, show the underlying\n> + ref pointed by it when showing a symbolic ref.\n> + peel: Show peeled tags.\n> + ref-pattern : When specified, only references matching the\n> + given patterns are displayed.\n\nWhat kind of pattern matching is allowed here?\nstrictly prefix only, or globbing, regexes?\nIs there a given grammar to follow? Maybe a link to the git\nglossary is or somewhere else might be fine.\n\nSeeing that we do wildmatch() down there (as opposed to regexes),\nI wonder if it provides an entry for a denial of service attack, by crafting\na pattern that is very expensive for the server to compute but cheap to\nask for from a client. (c.f. 94da9193a6 (grep: add support for PCRE v2,\n2017-06-01, but that is regexes!)\n\n> +The output of ls-refs is as follows:\n> +\n> + output = *ref\n> + flush-pkt\n> + ref = PKT-LINE((tip | peeled) LF)\n> + tip = obj-id SP refname (SP symref-target)\n> + peeled = obj-id SP refname \"^{}\"\n> +\n> + symref = PKT-LINE(\"symref\" SP symbolic-ref SP resolved-ref LF)\n> + shallow = PKT-LINE(\"shallow\" SP obj-id LF)\n> diff --git a/Makefile b/Makefile\n> index 5f3b5fe8b..152a73bec 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -820,6 +820,7 @@ LIB_OBJS += list-objects-filter-options.o\n> LIB_OBJS += ll-merge.o\n> LIB_OBJS += lockfile.o\n> LIB_OBJS += log-tree.o\n> +LIB_OBJS += ls-refs.o\n> LIB_OBJS += mailinfo.o\n> LIB_OBJS += mailmap.o\n> LIB_OBJS += match-trees.o\n> diff --git a/ls-refs.c b/ls-refs.c\n> new file mode 100644\n> index 000000000..ac4904a40\n> --- /dev/null\n> +++ b/ls-refs.c\n> @@ -0,0 +1,97 @@\n> +#include \"cache.h\"\n> +#include \"repository.h\"\n> +#include \"refs.h\"\n> +#include \"remote.h\"\n> +#include \"argv-array.h\"\n> +#include \"ls-refs.h\"\n> +#include \"pkt-line.h\"\n> +\n> +struct ls_refs_data {\n> + unsigned peel;\n> + unsigned symrefs;\n> + struct argv_array patterns;\n> +};\n> +\n> +/*\n> + * Check if one of the patterns matches the tail part of the ref.\n> + * If no patterns were provided, all refs match.\n> + */\n> +static int ref_match(const struct argv_array *patterns, const char *refname)\n> +{\n> + char *pathbuf;\n> + int i;\n> +\n> + if (!patterns->argc)\n> + return 1; /* no restriction */\n> +\n> + pathbuf = xstrfmt(\"/%s\", refname);\n> + for (i = 0; i < patterns->argc; i++) {\n> + if (!wildmatch(patterns->argv[i], pathbuf, 0)) {\n> + free(pathbuf);\n> + return 1;\n> + }\n> + }\n> + free(pathbuf);\n> + return 0;\n> +}\n> +\n> +static int send_ref(const char *refname, const struct object_id *oid,\n> + int flag, void *cb_data)\n> +{\n> + struct ls_refs_data *data = cb_data;\n> + const char *refname_nons = strip_namespace(refname);\n> + struct strbuf refline = STRBUF_INIT;\n> +\n> + if (!ref_match(&data->patterns, refname))\n> + return 0;\n> +\n> + strbuf_addf(&refline, \"%s %s\", oid_to_hex(oid), refname_nons);\n> + if (data->symrefs && flag & REF_ISSYMREF) {\n> + struct object_id unused;\n> + const char *symref_target = resolve_ref_unsafe(refname, 0,\n> + &unused,\n> + &flag);\n> +\n> + if (!symref_target)\n> + die(\"'%s' is a symref but it is not?\", refname);\n> +\n> + strbuf_addf(&refline, \" %s\", symref_target);\n> + }\n> +\n> + strbuf_addch(&refline, '\\n');\n> +\n> + packet_write(1, refline.buf, refline.len);\n> + if (data->peel) {\n> + struct object_id peeled;\n> + if (!peel_ref(refname, &peeled))\n> + packet_write_fmt(1, \"%s %s^{}\\n\", oid_to_hex(&peeled),\n> + refname_nons);\n> + }\n> +\n> + strbuf_release(&refline);\n> + return 0;\n> +}\n> +\n> +int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)\n> +{\n> + int i;\n> + struct ls_refs_data data = { 0, 0, ARGV_ARRAY_INIT };\n> +\n> + for (i = 0; i < args->argc; i++) {\n> + const char *arg = args->argv[i];\n> + const char *out;\n> +\n> + if (!strcmp(\"peel\", arg))\n> + data.peel = 1;\n> + else if (!strcmp(\"symrefs\", arg))\n> + data.symrefs = 1;\n> + else if (skip_prefix(arg, \"ref-pattern \", &out))\n> + argv_array_pushf(&data.patterns, \"*/%s\", out);\n> + }\n> +\n> + head_ref_namespaced(send_ref, &data);\n> + for_each_namespaced_ref(send_ref, &data);\n> + packet_flush(1);\n> + argv_array_clear(&data.patterns);\n> + return 0;\n> +}\n> diff --git a/ls-refs.h b/ls-refs.h\n> new file mode 100644\n> index 000000000..9e4c57bfe\n> --- /dev/null\n> +++ b/ls-refs.h\n> @@ -0,0 +1,9 @@\n> +#ifndef LS_REFS_H\n> +#define LS_REFS_H\n> +\n> +struct repository;\n> +struct argv_array;\n> +extern int ls_refs(struct repository *r, struct argv_array *keys,\n> + struct argv_array *args);\n> +\n> +#endif /* LS_REFS_H */\n> diff --git a/serve.c b/serve.c\n> index da8127775..88d548410 100644\n> --- a/serve.c\n> +++ b/serve.c\n> @@ -4,6 +4,7 @@\n> #include \"pkt-line.h\"\n> #include \"version.h\"\n> #include \"argv-array.h\"\n> +#include \"ls-refs.h\"\n> #include \"serve.h\"\n>\n> static int always_advertise(struct repository *r,\n> @@ -44,6 +45,7 @@ struct protocol_capability {\n> static struct protocol_capability capabilities[] = {\n> { \"agent\", agent_advertise, NULL },\n> { \"stateless-rpc\", always_advertise, NULL },\n> + { \"ls-refs\", always_advertise, ls_refs },\n> };\n>\n> static void advertise_capabilities(void)\n> --\n> 2.15.1.620.gb9897f4670-goog\n>\n"},{"id":"335811","messageId":"CAGZ79kYVeKS26qMEiu=2tP0t18no9CtEoAMeBO-rSt+66=fnGQ@mail.gmail.com","threadId":"47524","inReplyTo":"20180103001828.205012-20-bmwill@google.com","subject":"Re: [PATCH 19/26] upload-pack: introduce fetch server command","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-04T01:07:07Z","receivedAt":"2018-01-04T01:07:14Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> Introduce the 'fetch' server command.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> Documentation/technical/protocol-v2.txt | 14 ++\n> serve.c | 2 +\n> upload-pack.c | 290 ++++++++++++++++++++++++++++++++\n> upload-pack.h | 9 +\n> 4 files changed, 315 insertions(+)\n> create mode 100644 upload-pack.h\n>\n> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> index 5f4d0e719..2a8e2f226 100644\n> --- a/Documentation/technical/protocol-v2.txt\n> +++ b/Documentation/technical/protocol-v2.txt\n> @@ -115,3 +115,17 @@ The output of ls-refs is as follows:\n>\n> symref = PKT-LINE(\"symref\" SP symbolic-ref SP resolved-ref LF)\n> shallow = PKT-LINE(\"shallow\" SP obj-id LF)\n> +\n> + Fetch\n> +-------\n> +\n> +Fetch will need to be a modified version of the v1 fetch protocol. Some\n> +potential areas for improvement are: Ref-in-want, CDN offloading,\n> +Fetch-options.\n> +\n> +Since we'll have an 'ls-ref' service we can eliminate the need of fetch\n> +to perform a ref-advertisement, instead a client can run the 'ls-refs'\n> +service first, in order to find out what refs the server has, and then\n> +request those refs directly using the fetch service.\n> +\n> +//TODO Flesh out the design\n\nTODO: actually do it. ;)\n\na couple notes from the discussion in office:\n* Could we split fetch into multiple phases\n (negotiation + getting the pack)\n* negotiation could be reused in forced push to\n minimize the pack to be sent\n* negotiation in a half duplex is actually better\n called 'discovery', which discovers about the set\n of objects available on the remote side.\n (the opposite would be reveal, or 'ask-for-discovery', which\n is could be used for a symmetric design of fetch and push)\n\n\n> diff --git a/serve.c b/serve.c\n> index 88d548410..ca3bb7190 100644\n> --- a/serve.c\n> +++ b/serve.c\n> @@ -6,6 +6,7 @@\n> #include \"argv-array.h\"\n> #include \"ls-refs.h\"\n> #include \"serve.h\"\n> +#include \"upload-pack.h\"\n>\n> static int always_advertise(struct repository *r,\n> struct strbuf *value)\n> @@ -46,6 +47,7 @@ static struct protocol_capability capabilities[] = {\n> { \"agent\", agent_advertise, NULL },\n> { \"stateless-rpc\", always_advertise, NULL },\n> { \"ls-refs\", always_advertise, ls_refs },\n> + { \"fetch\", always_advertise, upload_pack_v2 },\n> };\n>\n> static void advertise_capabilities(void)\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 2ca60d27c..c41f6f528 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -20,6 +20,7 @@\n> #include \"prio-queue.h\"\n> #include \"protocol.h\"\n> #include \"serve.h\"\n> +#include \"upload-pack.h\"\n>\n> static const char * const upload_pack_usage[] = {\n> N_(\"git upload-pack [] \"),\n> @@ -1040,6 +1041,295 @@ static void upload_pack(void)\n> }\n> }\n>\n> +struct upload_pack_data {\n> + struct object_array wants;\n> + struct oid_array haves;\n> +\n> + unsigned stateless_rpc : 1;\n> +\n> + unsigned use_thin_pack : 1;\n> + unsigned use_ofs_delta : 1;\n> + unsigned no_progress : 1;\n> + unsigned use_include_tag : 1;\n> + unsigned done : 1;\n> +};\n> +\n> +#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }\n> +\n> +static void upload_pack_data_clear(struct upload_pack_data *data)\n> +{\n> + object_array_clear(&data->wants);\n> + oid_array_clear(&data->haves);\n> +}\n> +\n> +static int parse_want(const char *line)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"want \", &arg)) {\n> + struct object_id oid;\n> + struct object *o;\n> +\n> + if (get_oid_hex(arg, &oid))\n> + die(\"git upload-pack: protocol error, \"\n> + \"expected to get oid, not '%s'\", line);\n> +\n> + o = parse_object(&oid);\n> + if (!o) {\n> + packet_write_fmt(1,\n> + \"ERR upload-pack: not our ref %s\",\n> + oid_to_hex(&oid));\n> + die(\"git upload-pack: not our ref %s\",\n> + oid_to_hex(&oid));\n> + }\n> +\n> + if (!(o->flags & WANTED)) {\n> + o->flags |= WANTED;\n> + add_object_array(o, NULL, &want_obj);\n> + }\n> +\n> + return 1;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +static int parse_have(const char *line, struct oid_array *haves)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"have \", &arg)) {\n> + struct object_id oid;\n> +\n> + if (get_oid_hex(arg, &oid))\n> + die(\"git upload-pack: expected SHA1 object, got '%s'\", arg);\n> + oid_array_append(haves, &oid);\n> + return 1;\n> + }\n\nThe same stylistic nit as in an earlier patch: Maybe\nshortcircuit by inverting the condition?\n\n\n\n> +\n> + return 0;\n> +}\n> +\n> +static void process_args(struct argv_array *args, struct upload_pack_data *data)\n> +{\n> + int i;\n> +\n> + for (i = 0; i < args->argc; i++) {\n> + const char *arg = args->argv[i];\n> +\n> + /* process want */\n> + if (parse_want(arg))\n> + continue;\n> + /* process have line */\n> + if (parse_have(arg, &data->haves))\n> + continue;\n> +\n> + /* process args like thin-pack */\n> + if (!strcmp(arg, \"thin-pack\")) {\n> + use_thin_pack = 1;\n> + continue;\n> + }\n> + if (!strcmp(arg, \"ofs-delta\")) {\n> + use_ofs_delta = 1;\n> + continue;\n> + }\n> + if (!strcmp(arg, \"no-progress\")) {\n> + no_progress = 1;\n> + continue;\n> + }\n> + if (!strcmp(arg, \"include-tag\")) {\n> + use_include_tag = 1;\n> + continue;\n> + }\n> + if (!strcmp(arg, \"done\")) {\n> + data->done = 1;\n> + continue;\n> + }\n> +\n> + /* ignore unknown lines maybe? */\n> + die(\"unexpect line: '%s'\", arg);\n> + }\n> +}\n> +\n> +static void read_haves(struct upload_pack_data *data)\n> +{\n> + struct packet_reader reader;\n> + packet_reader_init(&reader, 0, NULL, 0,\n> + PACKET_READ_CHOMP_NEWLINE);\n> +\n> + while (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n> +\n> + if (parse_have(reader.line, &data->haves))\n> + continue;\n> + if (!strcmp(reader.line, \"done\")) {\n> + data->done = 1;\n> + continue;\n> + }\n> + }\n> + if (reader.status != PACKET_READ_FLUSH)\n> + die(\"ERROR\");\n> +}\n> +\n> +static int process_haves(struct oid_array *haves, struct oid_array *common)\n> +{\n> + int i;\n> +\n> + /* Process haves */\n> + for (i = 0; i < haves->nr; i++) {\n> + const struct object_id *oid = &haves->oid[i];\n> + struct object *o;\n> + int we_knew_they_have = 0;\n> +\n> + if (!has_object_file(oid))\n> + continue;\n> +\n> + oid_array_append(common, oid);\n> +\n> + o = parse_object(oid);\n> + if (!o)\n> + die(\"oops (%s)\", oid_to_hex(oid));\n> + if (o->type == OBJ_COMMIT) {\n> + struct commit_list *parents;\n> + struct commit *commit = (struct commit *)o;\n> + if (o->flags & THEY_HAVE)\n> + we_knew_they_have = 1;\n> + else\n> + o->flags |= THEY_HAVE;\n> + if (!oldest_have || (commit->date < oldest_have))\n> + oldest_have = commit->date;\n> + for (parents = commit->parents;\n> + parents;\n> + parents = parents->next)\n> + parents->item->object.flags |= THEY_HAVE;\n> + }\n> + if (!we_knew_they_have)\n> + add_object_array(o, NULL, &have_obj);\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +static int send_acks(struct oid_array *acks, struct strbuf *response)\n> +{\n> + int i;\n> + /* Send Acks */\n> + if (!acks->nr)\n> + packet_buf_write(response, \"NAK\\n\");\n> +\n> + for (i = 0; i < acks->nr; i++) {\n> + packet_buf_write(response, \"ACK %s common\\n\",\n> + oid_to_hex(&acks->oid[i]));\n> + }\n> +\n> + if (ok_to_give_up()) {\n> + /* Send Ready */\n> + packet_buf_write(response, \"ACK %s ready\\n\",\n> + oid_to_hex(&acks->oid[i-1]));\n> + return 1;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +static int process_haves_and_send_acks(struct upload_pack_data *data)\n> +{\n> + struct oid_array common = OID_ARRAY_INIT;\n> + struct strbuf response = STRBUF_INIT;\n> + int ret = 0;\n> +\n> + process_haves(&data->haves, &common);\n> + if (data->done) {\n> + ret = 1;\n> + } else if (send_acks(&common, &response)) {\n> + packet_buf_delim(&response);\n> + ret = 1;\n> + } else {\n> + /* Add Flush */\n> + packet_buf_flush(&response);\n> + ret = 0;\n> + }\n> +\n> + /* Send response */\n> + write_or_die(1, response.buf, response.len);\n> + strbuf_release(&response);\n> +\n> + oid_array_clear(&data->haves);\n> + oid_array_clear(&common);\n> + return ret;\n> +}\n> +\n> +#define FETCH_PROCESS_ARGS 0\n> +#define FETCH_READ_HAVES 1\n> +#define FETCH_SEND_ACKS 2\n> +#define FETCH_SEND_PACK 3\n> +#define FETCH_DONE 4\n> +\n> +int upload_pack_v2(struct repository *r, struct argv_array *keys,\n> + struct argv_array *args)\n> +{\n> + int state = FETCH_PROCESS_ARGS;\n> + struct upload_pack_data data = UPLOAD_PACK_DATA_INIT;\n> + const char *out;\n> + use_sideband = LARGE_PACKET_MAX;\n> +\n> + /* Check if cmd is being run as a stateless-rpc */\n> + if (has_capability(keys, \"stateless-rpc\", &out))\n> + if (!strcmp(out, \"true\"))\n> + data.stateless_rpc = 1;\n> +\n> + while (state != FETCH_DONE) {\n> + switch (state) {\n> + case FETCH_PROCESS_ARGS:\n> + process_args(args, &data);\n> +\n> + if (!want_obj.nr) {\n> + /*\n> + * Request didn't contain any 'want' lines,\n> + * guess they didn't want anything.\n> + */\n> + state = FETCH_DONE;\n> + } else if (data.haves.nr) {\n> + /*\n> + * Request had 'have' lines, so lets ACK them.\n> + */\n> + state = FETCH_SEND_ACKS;\n> + } else {\n> + /*\n> + * Request had 'want's but no 'have's so we can\n> + * immedietly go to construct and send a pack.\n> + */\n> + state = FETCH_SEND_PACK;\n> + }\n> + break;\n> + case FETCH_READ_HAVES:\n> + read_haves(&data);\n> + state = FETCH_SEND_ACKS;\n> + break;\n> + case FETCH_SEND_ACKS:\n> + if (process_haves_and_send_acks(&data))\n> + state = FETCH_SEND_PACK;\n> + else if (data.stateless_rpc)\n> + /*\n> + * Request was made via stateless-rpc and a\n> + * packfile isn't ready to be created and sent.\n> + */\n> + state = FETCH_DONE;\n> + else\n> + state = FETCH_READ_HAVES;\n> + break;\n> + case FETCH_SEND_PACK:\n> + create_pack_file();\n> + state = FETCH_DONE;\n> + break;\n> + case FETCH_DONE:\n> + break;\n> + default:\n> + BUG(\"invalid state\");\n> + }\n> + }\n> +\n> + upload_pack_data_clear(&data);\n> + return 0;\n> +}\n> +\n> static int upload_pack_config(const char *var, const char *value, void *unused)\n> {\n> if (!strcmp(\"uploadpack.allowtipsha1inwant\", var)) {\n> diff --git a/upload-pack.h b/upload-pack.h\n> new file mode 100644\n> index 000000000..54c429563\n> --- /dev/null\n> +++ b/upload-pack.h\n> @@ -0,0 +1,9 @@\n> +#ifndef UPLOAD_PACK_H\n> +#define UPLOAD_PACK_H\n> +\n> +struct repository;\n> +struct argv_array;\n> +extern int upload_pack_v2(struct repository *r, struct argv_array *keys,\n> + struct argv_array *args);\n> +\n> +#endif /* UPLOAD_PACK_H */\n> --\n> 2.15.1.620.gb9897f4670-goog\n>\n"},{"id":"335812","messageId":"CAGZ79kbM4Jk=aHJu7+U3AcgFMh0-jbbu4RPLUkKwM0mDWge+gA@mail.gmail.com","threadId":"47524","inReplyTo":"20180103001828.205012-21-bmwill@google.com","subject":"Re: [PATCH 20/26] fetch-pack: perform a fetch using v2","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-04T01:23:08Z","receivedAt":"2018-01-04T01:23:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> When communicating with a v2 server, perform a fetch by requesting the\n> 'fetch' command.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> builtin/fetch-pack.c | 2 +-\n> fetch-pack.c | 267 ++++++++++++++++++++++++++++++++++++++++++++++++-\n> fetch-pack.h | 4 +-\n> t/t5701-protocol-v2.sh | 40 ++++++++\n> transport.c | 8 +-\n> 5 files changed, 314 insertions(+), 7 deletions(-)\n>\n> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n> index f492e8abd..867dd3cc7 100644\n> --- a/builtin/fetch-pack.c\n> +++ b/builtin/fetch-pack.c\n> @@ -213,7 +213,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> }\n>\n> ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n> - &shallow, pack_lockfile_ptr);\n> + &shallow, pack_lockfile_ptr, protocol_v0);\n> if (pack_lockfile) {\n> printf(\"lock %s\\n\", pack_lockfile);\n> fflush(stdout);\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 9f6b07ad9..c26fdc539 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -1008,6 +1008,262 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n> return ref;\n> }\n>\n> +static void add_wants(const struct ref *wants, struct strbuf *req_buf)\n> +{\n> + for ( ; wants ; wants = wants->next) {\n> + const struct object_id *remote = &wants->old_oid;\n> + const char *remote_hex;\n> + struct object *o;\n> +\n> + /*\n> + * If that object is complete (i.e. it is an ancestor of a\n> + * local ref), we tell them we have it but do not have to\n> + * tell them about its ancestors, which they already know\n> + * about.\n> + *\n> + * We use lookup_object here because we are only\n> + * interested in the case we *know* the object is\n> + * reachable and we have already scanned it.\n> + */\n> + if (((o = lookup_object(remote->hash)) != NULL) &&\n> + (o->flags & COMPLETE)) {\n> + continue;\n> + }\n> +\n> + remote_hex = oid_to_hex(remote);\n> + packet_buf_write(req_buf, \"want %s\\n\", remote_hex);\n> + }\n> +}\n> +\n> +static int add_haves(struct strbuf *req_buf, int *in_vain)\n> +{\n> + int ret = 0;\n> + int haves_added = 0;\n> + const struct object_id *oid;\n> +\n> + while ((oid = get_rev())) {\n> + packet_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n> + if (++haves_added >= INITIAL_FLUSH)\n> + break;\n> + };\n> +\n> + *in_vain += haves_added;\n> + if (!haves_added || *in_vain >= MAX_IN_VAIN) {\n> + /* Send Done */\n> + packet_buf_write(req_buf, \"done\\n\");\n> + ret = 1;\n> + }\n> +\n> + return ret;\n> +}\n> +\n> +static int send_haves(int fd_out, int *in_vain)\n> +{\n> + int ret = 0;\n> + struct strbuf req_buf = STRBUF_INIT;\n> +\n> + ret = add_haves(&req_buf, in_vain);\n> +\n> + /* Send request */\n> + packet_buf_flush(&req_buf);\n> + write_or_die(fd_out, req_buf.buf, req_buf.len);\n> +\n> + strbuf_release(&req_buf);\n> + return ret;\n> +}\n> +\n> +static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n> + const struct ref *wants, struct oidset *common,\n> + int *in_vain)\n> +{\n> + int ret = 0;\n> + struct strbuf req_buf = STRBUF_INIT;\n> +\n> + packet_buf_write(&req_buf, \"command=fetch\");\n> + packet_buf_write(&req_buf, \"agent=%s\", git_user_agent_sanitized());\n> + if (args->stateless_rpc)\n> + packet_buf_write(&req_buf, \"stateless-rpc=true\");\n> +\n> + packet_buf_delim(&req_buf);\n> + if (args->use_thin_pack)\n> + packet_buf_write(&req_buf, \"thin-pack\");\n> + if (args->no_progress)\n> + packet_buf_write(&req_buf, \"no-progress\");\n> + if (args->include_tag)\n> + packet_buf_write(&req_buf, \"include-tag\");\n> + if (prefer_ofs_delta)\n> + packet_buf_write(&req_buf, \"ofs-delta\");\n> +\n> + /* add wants */\n> + add_wants(wants, &req_buf);\n> +\n> + /*\n> + * If we are running stateless-rpc we need to add all the common\n> + * commits we've found in previous rounds\n> + */\n> + if (args->stateless_rpc) {\n> + struct oidset_iter iter;\n> + const struct object_id *oid;\n> + oidset_iter_init(common, &iter);\n> +\n> + while ((oid = oidset_iter_next(&iter))) {\n> + packet_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n> + }\n> + }\n> +\n> + /* Add initial haves */\n> + ret = add_haves(&req_buf, in_vain);\n> +\n> + /* Send request */\n> + packet_buf_flush(&req_buf);\n> + write_or_die(fd_out, req_buf.buf, req_buf.len);\n> +\n> + strbuf_release(&req_buf);\n> + return ret;\n> +}\n> +\n> +static enum ack_type process_ack(const char *line, struct object_id *oid)\n> +{\n> + const char *arg;\n> +\n> + if (!strcmp(line, \"NAK\"))\n> + return NAK;\n> + if (skip_prefix(line, \"ACK \", &arg)) {\n> + if (!parse_oid_hex(arg, oid, &arg)) {\n> + if (strstr(arg, \"continue\"))\n> + return ACK_continue;\n> + if (strstr(arg, \"common\"))\n> + return ACK_common;\n> + if (strstr(arg, \"ready\"))\n> + return ACK_ready;\n> + return ACK;\n> + }\n> + }\n> + if (skip_prefix(line, \"ERR \", &arg))\n> + die(_(\"remote error: %s\"), arg);\n> + die(_(\"git fetch-pack: expected ACK/NAK, got '%s'\"), line);\n> +}\n> +\n> +static int process_acks(struct packet_reader *reader, struct oidset *common)\n> +{\n> + int got_ready = 0;\n> + int got_common = 0;\n> + while (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n> + struct object_id oid;\n> + struct commit *commit;\n> + enum ack_type ack = process_ack(reader->line, &oid);\n> +\n> + switch (ack) {\n> + case ACK_ready:\n> + clear_prio_queue(&rev_list);\n> + got_ready = 1;\n> + /* fallthrough */\n> + case ACK_common:\n> + oidset_insert(common, &oid);\n> + commit = lookup_commit(&oid);\n> + mark_common(commit, 0, 1);\n> + got_common = 1;\n> + break;\n> + case NAK:\n> + break;\n> + case ACK:\n> + case ACK_continue:\n> + die(\"ACK/ACK_continue not supported\");\n> + }\n> + }\n> +\n> + if (reader->status != PACKET_READ_FLUSH &&\n> + reader->status != PACKET_READ_DELIM)\n> + die(\"Error during processing acks: %d\", reader->status);\n> +\n> + /* return 0 if no common, 1 if there are common, or 2 if ready */\n> + return got_ready + got_common;\n> +}\n> +\n> +#define FETCH_CHECK_LOCAL 0\n> +#define FETCH_SEND_REQUEST 1\n> +#define FETCH_PROCESS_ACKS 2\n> +#define FETCH_SEND_HAVES 3\n> +#define FETCH_GET_PACK 4\n> +#define FETCH_DONE 5\n> +\n> +static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> + int fd[2],\n> + const struct ref *orig_ref,\n> + struct ref **sought, int nr_sought,\n> + char **pack_lockfile)\n> +{\n> + struct ref *ref = copy_ref_list(orig_ref);\n> + int state = FETCH_CHECK_LOCAL;\n\nIs there any reason to use #defines over an enum here?\n\n> + struct oidset common = OIDSET_INIT;\n> + struct packet_reader reader;\n> + int in_vain = 0;\n> + packet_reader_init(&reader, fd[0], NULL, 0,\n> + PACKET_READ_CHOMP_NEWLINE);\n> +\n> + while (state != FETCH_DONE) {\n> + switch (state) {\n> + case FETCH_CHECK_LOCAL:\n> + sort_ref_list(&ref, ref_compare_name);\n> + QSORT(sought, nr_sought, cmp_ref_by_name);\n> +\n> + /* v2 supports these by default */\n> + allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n> + use_sideband = 2;\n> +\n> + /* Filter 'ref' by 'sought' and those that aren't local */\n> + if (everything_local(args, &ref, sought, nr_sought))\n> + state = FETCH_DONE;\n> + else\n> + state = FETCH_SEND_REQUEST;\n> + break;\n> + case FETCH_SEND_REQUEST:\n> + if (send_fetch_request(fd[1], args, ref, &common, &in_vain))\n> + state = FETCH_GET_PACK;\n> + else\n> + state = FETCH_PROCESS_ACKS;\n> + break;\n> + case FETCH_PROCESS_ACKS:\n> + /* Process ACKs/NAKs */\n> + switch (process_acks(&reader, &common)) {\n> + case 2:\n> + state = FETCH_GET_PACK;\n> + break;\n> + case 1:\n> + in_vain = 0;\n> + /* fallthrough */\n> + default:\n> + if (args->stateless_rpc)\n> + state = FETCH_SEND_REQUEST;\n> + else\n> + state = FETCH_SEND_HAVES;\n> + break;\n> + }\n> + break;\n> + case FETCH_SEND_HAVES:\n> + if (send_haves(fd[1], &in_vain))\n> + state = FETCH_GET_PACK;\n> + else\n> + state = FETCH_PROCESS_ACKS;\n> + break;\n> + case FETCH_GET_PACK:\n> + /* get the pack */\n> + if (get_pack(args, fd, pack_lockfile))\n> + die(_(\"git fetch-pack: fetch failed.\"));\n> +\n> + state = FETCH_DONE;\n> + break;\n> + case FETCH_DONE:\n> + break;\n> + default:\n> + die(\"invalid state\");\n> + }\n> + }\n> +\n> + oidset_clear(&common);\n> + return ref;\n> +}\n> +\n> static void fetch_pack_config(void)\n> {\n> git_config_get_int(\"fetch.unpacklimit\", &fetch_unpack_limit);\n> @@ -1153,7 +1409,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n> const char *dest,\n> struct ref **sought, int nr_sought,\n> struct oid_array *shallow,\n> - char **pack_lockfile)\n> + char **pack_lockfile,\n> + enum protocol_version version)\n> {\n> struct ref *ref_cpy;\n> struct shallow_info si;\n> @@ -1167,8 +1424,12 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n> die(_(\"no matching remote head\"));\n> }\n> prepare_shallow_info(&si, shallow);\n> - ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n> - &si, pack_lockfile);\n> + if (version == protocol_v2)\n> + ref_cpy = do_fetch_pack_v2(args, fd, ref, sought, nr_sought,\n> + pack_lockfile);\n> + else\n> + ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n> + &si, pack_lockfile);\n> reprepare_packed_git();\n> update_shallow(args, sought, nr_sought, &si);\n> clear_shallow_info(&si);\n> diff --git a/fetch-pack.h b/fetch-pack.h\n> index b6aeb43a8..7afca7305 100644\n> --- a/fetch-pack.h\n> +++ b/fetch-pack.h\n> @@ -3,6 +3,7 @@\n>\n> #include \"string-list.h\"\n> #include \"run-command.h\"\n> +#include \"protocol.h\"\n>\n> struct oid_array;\n>\n> @@ -43,7 +44,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n> struct ref **sought,\n> int nr_sought,\n> struct oid_array *shallow,\n> - char **pack_lockfile);\n> + char **pack_lockfile,\n> + enum protocol_version version);\n>\n> /*\n> * Print an appropriate error message for each sought ref that wasn't\n> diff --git a/t/t5701-protocol-v2.sh b/t/t5701-protocol-v2.sh\n> index 7d8aeb766..3e411e178 100755\n> --- a/t/t5701-protocol-v2.sh\n> +++ b/t/t5701-protocol-v2.sh\n> @@ -33,4 +33,44 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol\n> ! grep \"refs/tags/\" log\n> '\n>\n> +test_expect_success 'clone with file:// using protocol v2' '\n> + GIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n> + clone \"file://$(pwd)/file_parent\" file_child 2>log &&\n> +\n> + git -C file_child log -1 --format=%s >actual &&\n> + git -C file_parent log -1 --format=%s >expect &&\n> + test_cmp expect actual &&\n> +\n> + # Server responded using protocol v1\n> + grep \"clone< version 2\" log\n> +'\n> +\n> +test_expect_success 'fetch with file:// using protocol v2' '\n> + test_commit -C file_parent two &&\n> +\n> + GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n> + fetch origin 2>log &&\n> +\n> + git -C file_child log -1 --format=%s origin/master >actual &&\n> + git -C file_parent log -1 --format=%s >expect &&\n> + test_cmp expect actual &&\n> +\n> + # Server responded using protocol v1\n> + grep \"fetch< version 2\" log\n> +'\n> +\n> +test_expect_success 'ref advertisment is filtered during fetch using protocol v2' '\n> + test_commit -C file_parent three &&\n> +\n> + GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n> + fetch origin master 2>log &&\n> +\n> + git -C file_child log -1 --format=%s origin/master >actual &&\n> + git -C file_parent log -1 --format=%s >expect &&\n> + test_cmp expect actual &&\n> +\n> + grep \"ref-pattern master\" log &&\n> + ! grep \"refs/tags/\" log\n> +'\n> +\n> test_done\n> diff --git a/transport.c b/transport.c\n> index 6ea3905e3..4fdbd9adc 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -256,14 +256,18 @@ static int fetch_refs_via_pack(struct transport *transport,\n>\n> switch (data->version) {\n> case protocol_v2:\n> - die(\"support for protocol v2 not implemented yet\");\n> + refs = fetch_pack(&args, data->fd, data->conn,\n> + refs_tmp ? refs_tmp : transport->remote_refs,\n> + dest, to_fetch, nr_heads, &data->shallow,\n> + &transport->pack_lockfile, data->version);\n> + packet_flush(data->fd[1]);\n> break;\n> case protocol_v1:\n> case protocol_v0:\n> refs = fetch_pack(&args, data->fd, data->conn,\n> refs_tmp ? refs_tmp : transport->remote_refs,\n> dest, to_fetch, nr_heads, &data->shallow,\n> - &transport->pack_lockfile);\n> + &transport->pack_lockfile, data->version);\n> break;\n> case protocol_unknown_version:\n> BUG(\"unknown protocol version\");\n> --\n> 2.15.1.620.gb9897f4670-goog\n>\n"},{"id":"336030","messageId":"20180105234108.GA172942@google.com","threadId":"47524","inReplyTo":"CAGZ79kbd1_nnYEJVN8a_NuwrEfHbT0wiwEfdkUz-kZRdpstqHA@mail.gmail.com","subject":"Re: [PATCH 01/26] pkt-line: introduce packet_read_with_status","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-05T23:41:08Z","receivedAt":"2018-01-05T23:41:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/03, Stefan Beller wrote:\n> On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> > The current pkt-line API encodes the status of a pkt-line read in the\n> > length of the read content. An error is indicated with '-1', a flush\n> > with '0' (which can be confusing since a return value of '0' can also\n> > indicate an empty pkt-line), and a positive integer for the length of\n> > the read content otherwise. This doesn't leave much room for allowing\n> > the addition of additional special packets in the future.\n> >\n> > To solve this introduce 'packet_read_with_status()' which reads a packet\n> > and returns the status of the read encoded as an 'enum packet_status'\n> > type. This allows for easily identifying between special and normal\n> > packets as well as errors. It also enables easily adding a new special\n> > packet in the future.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > pkt-line.c | 55 ++++++++++++++++++++++++++++++++++++++++++-------------\n> > pkt-line.h | 15 +++++++++++++++\n> > 2 files changed, 57 insertions(+), 13 deletions(-)\n> >\n> > diff --git a/pkt-line.c b/pkt-line.c\n> > index 2827ca772..8d7cd389f 100644\n> > --- a/pkt-line.c\n> > +++ b/pkt-line.c\n> > @@ -280,28 +280,33 @@ static int packet_length(const char *linelen)\n> > return (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);\n> > }\n> >\n> > -int packet_read(int fd, char **src_buf, size_t *src_len,\n> > - char *buffer, unsigned size, int options)\n> > +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n> > + char *buffer, unsigned size, int *pktlen,\n> > + int options)\n> > {\n> > - int len, ret;\n> > + int len;\n> > char linelen[4];\n> >\n> > - ret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);\n> > - if (ret < 0)\n> > - return ret;\n> > + if (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)\n> > + return PACKET_READ_EOF;\n> > +\n> > len = packet_length(linelen);\n> > if (len < 0)\n> > die(\"protocol error: bad line length character: %.4s\", linelen);\n> > - if (!len) {\n> > +\n> > + if (len == 0) {\n> > packet_trace(\"0000\", 4, 0);\n> > - return 0;\n> > + return PACKET_READ_FLUSH;\n> > + } else if (len >= 1 && len <= 3) {\n> > + die(\"protocol error: bad line length character: %.4s\", linelen);\n> \n> I wonder how much libified code we want here already, maybe we could\n> have PACKET_READ_ERROR as a return value here instead of die()ing.\n> There could also be an option that tells this code to die on error, this reminds\n> me of the repository discovery as well as the refs code, both of which have\n> this pattern.\n> \n> Currently this series is only upgrading commands that use the network\n> anyway, so I guess die()ing in an ls-remote or fetch is no big deal,\n> but it could\n> be interesting to keep going once we have more of the partial clone\n> stuff working\n> (e.g. remote assisted log/blame would want to gracefully fall back instead of\n> die()ing without any useful output, I would think.)\n\nThese are all things we could do, but the current code just dies and it\nmay be more hassle right now to change all the uses of packet_read to\nhandle errors gracefully. But its definitely something we can do in the\nfuture.\n\n> \n> > }\n> > +\n> > len -= 4;\n> > - if (len >= size)\n> > + if ((len < 0) || ((unsigned)len >= size))\n> > die(\"protocol error: bad line length %d\", len);\n> > - ret = get_packet_data(fd, src_buf, src_len, buffer, len, options);\n> > - if (ret < 0)\n> > - return ret;\n> > +\n> > + if (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)\n> > + return PACKET_READ_EOF;\n> >\n> > if ((options & PACKET_READ_CHOMP_NEWLINE) &&\n> > len && buffer[len-1] == '\\n')\n> > @@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,\n> >\n> > buffer[len] = 0;\n> > packet_trace(buffer, len, 0);\n> > - return len;\n> > + *pktlen = len;\n> > + return PACKET_READ_NORMAL;\n> > +}\n> > +\n> > +int packet_read(int fd, char **src_buffer, size_t *src_len,\n> > + char *buffer, unsigned size, int options)\n> > +{\n> > + enum packet_read_status status;\n> > + int pktlen;\n> > +\n> > + status = packet_read_with_status(fd, src_buffer, src_len,\n> > + buffer, size, &pktlen,\n> > + options);\n> > + switch (status) {\n> > + case PACKET_READ_EOF:\n> > + pktlen = -1;\n> > + break;\n> > + case PACKET_READ_NORMAL:\n> > + break;\n> > + case PACKET_READ_FLUSH:\n> > + pktlen = 0;\n> > + break;\n> > + }\n> > +\n> > + return pktlen;\n> > }\n> >\n> > static char *packet_read_line_generic(int fd,\n> > diff --git a/pkt-line.h b/pkt-line.h\n> > index 3dad583e2..06c468927 100644\n> > --- a/pkt-line.h\n> > +++ b/pkt-line.h\n> > @@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n> > int packet_read(int fd, char **src_buffer, size_t *src_len, char\n> > *buffer, unsigned size, int options);\n> >\n> > +/*\n> > + * Read a packetized line into a buffer like the 'packet_read()' function but\n> > + * returns an 'enum packet_read_status' which indicates the status of the read.\n> > + * The number of bytes read will be assigined to *pktlen if the status of the\n> > + * read was 'PACKET_READ_NORMAL'.\n> > + */\n> > +enum packet_read_status {\n> > + PACKET_READ_EOF = -1,\n> > + PACKET_READ_NORMAL,\n> > + PACKET_READ_FLUSH,\n> > +};\n> > +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n> > + char *buffer, unsigned size, int *pktlen,\n> > + int options);\n> > +\n> > /*\n> > * Convenience wrapper for packet_read that is not gentle, and sets the\n> > * CHOMP_NEWLINE option. The return value is NULL for a flush packet,\n> > --\n> > 2.15.1.620.gb9897f4670-goog\n> >\n\n-- \nBrandon Williams\n"},{"id":"336031","messageId":"20180105234902.GB172942@google.com","threadId":"47524","inReplyTo":"CAGZ79kaQMBaA+gxqjY0xBNZiy2PdafoXjB+kowM9pj9heLwRPA@mail.gmail.com","subject":"Re: [PATCH 12/26] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-05T23:49:02Z","receivedAt":"2018-01-05T23:49:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/03, Stefan Beller wrote:\n> On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> > Introduce the ls-refs server command. In protocol v2, the ls-refs\n> > command is used to request the ref advertisement from the server. Since\n> > it is a command which can be requested (as opposed to mandatory in v1),\n> > a client can sent a number of parameters in its request to limit the ref\n> > advertisement based on provided ref-patterns.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > Documentation/technical/protocol-v2.txt | 26 +++++++++\n> > Makefile | 1 +\n> > ls-refs.c | 97 +++++++++++++++++++++++++++++++++\n> > ls-refs.h | 9 +++\n> \n> Maybe consider putting any served command into a sub directory?\n> \n> For example the code in builtin/ has laxer rules w.r.t. die()ing\n> as it is a user facing command, whereas some devs want to see\n> code at the root of the repo to not die() at all as the eventual goal\n> is to have a library there.\n> All this code is on the remote side, which also has different traits than\n> the code at the root of the git.git repo; non-localisation comes to mind,\n> but there might be other aspects as well (security?).\n\nWell if we were to do this then we should move upload-pack and\nreceive-pack into this same \"server code\" directory.\n\n> \n> \n> > serve.c | 2 +\n> > 5 files changed, 135 insertions(+)\n> > create mode 100644 ls-refs.c\n> > create mode 100644 ls-refs.h\n> >\n> > diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> > index b87ba3816..5f4d0e719 100644\n> > --- a/Documentation/technical/protocol-v2.txt\n> > +++ b/Documentation/technical/protocol-v2.txt\n> > @@ -89,3 +89,29 @@ terminate the connection.\n> > Commands are the core actions that a client wants to perform (fetch, push,\n> > etc). Each command will be provided with a list capabilities and\n> > arguments as requested by a client.\n> > +\n> > + Ls-refs\n> \n> So is it ls-refs or Ls-refs or is any capitalization valid?\n\n\"ls-refs\" I'll make sure to change this.\n> \n> > +---------\n> > +\n> > +Ls-refs is the command used to request a reference advertisement in v2.\n> > +Unlike the current reference advertisement, ls-refs takes in parameters\n> > +which can be used to limit the refs sent from the server.\n> > +\n> > +Ls-ref takes in the following parameters wraped in packet-lines:\n> > +\n> > + symrefs: In addition to the object pointed by it, show the underlying\n> > + ref pointed by it when showing a symbolic ref.\n> > + peel: Show peeled tags.\n> > + ref-pattern : When specified, only references matching the\n> > + given patterns are displayed.\n> \n> What kind of pattern matching is allowed here?\n> strictly prefix only, or globbing, regexes?\n> Is there a given grammar to follow? Maybe a link to the git\n> glossary is or somewhere else might be fine.\n> \n> Seeing that we do wildmatch() down there (as opposed to regexes),\n> I wonder if it provides an entry for a denial of service attack, by crafting\n> a pattern that is very expensive for the server to compute but cheap to\n> ask for from a client. (c.f. 94da9193a6 (grep: add support for PCRE v2,\n> 2017-06-01, but that is regexes!)\n> \n> > +The output of ls-refs is as follows:\n> > +\n> > + output = *ref\n> > + flush-pkt\n> > + ref = PKT-LINE((tip | peeled) LF)\n> > + tip = obj-id SP refname (SP symref-target)\n> > + peeled = obj-id SP refname \"^{}\"\n> > +\n> > + symref = PKT-LINE(\"symref\" SP symbolic-ref SP resolved-ref LF)\n> > + shallow = PKT-LINE(\"shallow\" SP obj-id LF)\n> > diff --git a/Makefile b/Makefile\n> > index 5f3b5fe8b..152a73bec 100644\n> > --- a/Makefile\n> > +++ b/Makefile\n> > @@ -820,6 +820,7 @@ LIB_OBJS += list-objects-filter-options.o\n> > LIB_OBJS += ll-merge.o\n> > LIB_OBJS += lockfile.o\n> > LIB_OBJS += log-tree.o\n> > +LIB_OBJS += ls-refs.o\n> > LIB_OBJS += mailinfo.o\n> > LIB_OBJS += mailmap.o\n> > LIB_OBJS += match-trees.o\n> > diff --git a/ls-refs.c b/ls-refs.c\n> > new file mode 100644\n> > index 000000000..ac4904a40\n> > --- /dev/null\n> > +++ b/ls-refs.c\n> > @@ -0,0 +1,97 @@\n> > +#include \"cache.h\"\n> > +#include \"repository.h\"\n> > +#include \"refs.h\"\n> > +#include \"remote.h\"\n> > +#include \"argv-array.h\"\n> > +#include \"ls-refs.h\"\n> > +#include \"pkt-line.h\"\n> > +\n> > +struct ls_refs_data {\n> > + unsigned peel;\n> > + unsigned symrefs;\n> > + struct argv_array patterns;\n> > +};\n> > +\n> > +/*\n> > + * Check if one of the patterns matches the tail part of the ref.\n> > + * If no patterns were provided, all refs match.\n> > + */\n> > +static int ref_match(const struct argv_array *patterns, const char *refname)\n> > +{\n> > + char *pathbuf;\n> > + int i;\n> > +\n> > + if (!patterns->argc)\n> > + return 1; /* no restriction */\n> > +\n> > + pathbuf = xstrfmt(\"/%s\", refname);\n> > + for (i = 0; i < patterns->argc; i++) {\n> > + if (!wildmatch(patterns->argv[i], pathbuf, 0)) {\n> > + free(pathbuf);\n> > + return 1;\n> > + }\n> > + }\n> > + free(pathbuf);\n> > + return 0;\n> > +}\n> > +\n> > +static int send_ref(const char *refname, const struct object_id *oid,\n> > + int flag, void *cb_data)\n> > +{\n> > + struct ls_refs_data *data = cb_data;\n> > + const char *refname_nons = strip_namespace(refname);\n> > + struct strbuf refline = STRBUF_INIT;\n> > +\n> > + if (!ref_match(&data->patterns, refname))\n> > + return 0;\n> > +\n> > + strbuf_addf(&refline, \"%s %s\", oid_to_hex(oid), refname_nons);\n> > + if (data->symrefs && flag & REF_ISSYMREF) {\n> > + struct object_id unused;\n> > + const char *symref_target = resolve_ref_unsafe(refname, 0,\n> > + &unused,\n> > + &flag);\n> > +\n> > + if (!symref_target)\n> > + die(\"'%s' is a symref but it is not?\", refname);\n> > +\n> > + strbuf_addf(&refline, \" %s\", symref_target);\n> > + }\n> > +\n> > + strbuf_addch(&refline, '\\n');\n> > +\n> > + packet_write(1, refline.buf, refline.len);\n> > + if (data->peel) {\n> > + struct object_id peeled;\n> > + if (!peel_ref(refname, &peeled))\n> > + packet_write_fmt(1, \"%s %s^{}\\n\", oid_to_hex(&peeled),\n> > + refname_nons);\n> > + }\n> > +\n> > + strbuf_release(&refline);\n> > + return 0;\n> > +}\n> > +\n> > +int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)\n> > +{\n> > + int i;\n> > + struct ls_refs_data data = { 0, 0, ARGV_ARRAY_INIT };\n> > +\n> > + for (i = 0; i < args->argc; i++) {\n> > + const char *arg = args->argv[i];\n> > + const char *out;\n> > +\n> > + if (!strcmp(\"peel\", arg))\n> > + data.peel = 1;\n> > + else if (!strcmp(\"symrefs\", arg))\n> > + data.symrefs = 1;\n> > + else if (skip_prefix(arg, \"ref-pattern \", &out))\n> > + argv_array_pushf(&data.patterns, \"*/%s\", out);\n> > + }\n> > +\n> > + head_ref_namespaced(send_ref, &data);\n> > + for_each_namespaced_ref(send_ref, &data);\n> > + packet_flush(1);\n> > + argv_array_clear(&data.patterns);\n> > + return 0;\n> > +}\n> > diff --git a/ls-refs.h b/ls-refs.h\n> > new file mode 100644\n> > index 000000000..9e4c57bfe\n> > --- /dev/null\n> > +++ b/ls-refs.h\n> > @@ -0,0 +1,9 @@\n> > +#ifndef LS_REFS_H\n> > +#define LS_REFS_H\n> > +\n> > +struct repository;\n> > +struct argv_array;\n> > +extern int ls_refs(struct repository *r, struct argv_array *keys,\n> > + struct argv_array *args);\n> > +\n> > +#endif /* LS_REFS_H */\n> > diff --git a/serve.c b/serve.c\n> > index da8127775..88d548410 100644\n> > --- a/serve.c\n> > +++ b/serve.c\n> > @@ -4,6 +4,7 @@\n> > #include \"pkt-line.h\"\n> > #include \"version.h\"\n> > #include \"argv-array.h\"\n> > +#include \"ls-refs.h\"\n> > #include \"serve.h\"\n> >\n> > static int always_advertise(struct repository *r,\n> > @@ -44,6 +45,7 @@ struct protocol_capability {\n> > static struct protocol_capability capabilities[] = {\n> > { \"agent\", agent_advertise, NULL },\n> > { \"stateless-rpc\", always_advertise, NULL },\n> > + { \"ls-refs\", always_advertise, ls_refs },\n> > };\n> >\n> > static void advertise_capabilities(void)\n> > --\n> > 2.15.1.620.gb9897f4670-goog\n> >\n\n-- \nBrandon Williams\n"},{"id":"336032","messageId":"20180105235559.GC172942@google.com","threadId":"47524","inReplyTo":"CAGZ79kbM4Jk=aHJu7+U3AcgFMh0-jbbu4RPLUkKwM0mDWge+gA@mail.gmail.com","subject":"Re: [PATCH 20/26] fetch-pack: perform a fetch using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-05T23:55:59Z","receivedAt":"2018-01-05T23:56:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/03, Stefan Beller wrote:\n> On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n> > +\n> > +#define FETCH_CHECK_LOCAL 0\n> > +#define FETCH_SEND_REQUEST 1\n> > +#define FETCH_PROCESS_ACKS 2\n> > +#define FETCH_SEND_HAVES 3\n> > +#define FETCH_GET_PACK 4\n> > +#define FETCH_DONE 5\n> > +\n> > +static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> > + int fd[2],\n> > + const struct ref *orig_ref,\n> > + struct ref **sought, int nr_sought,\n> > + char **pack_lockfile)\n> > +{\n> > + struct ref *ref = copy_ref_list(orig_ref);\n> > + int state = FETCH_CHECK_LOCAL;\n> \n> Is there any reason to use #defines over an enum here?\n> \n\nNo, it would probably be better to use an enum, that would also get rid\nof the default case of the switch statement.\n\n-- \nBrandon Williams\n"},{"id":"336269","messageId":"20180109095534.c0c9b9ad3933d406c993c3ab@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"Re: [PATCH 00/26] protocol version 2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-09T17:55:34Z","receivedAt":"2018-01-09T17:55:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:02 -0800\nBrandon Williams wrote:\n\n> The following patches extend what I sent out as an WIP\n> (https://public-inbox.org/git/20171204235823.63299-1-bmwill@google.com/) and\n> implement protocol version 2.\n\nSummarizing (for myself) the rationale for protocol version 2:\n\nThe existing protocol has a few pain points: (a) limit on the length of\nthe capability line (the capability line can be used to include\nadditional parameters in a backwards-compatible way), (b) difficulty in\ncreating proxies because of inconsistent flush semantics, and (c) the\nneed to implement clients twice - once for HTTP and once for\nconnect-supporting transports. To which we can add another: (d) if we\nwant to support something entirely new (for example, a server-side \"git\nlog\"), we will need a new protocol anyway.\n\nThe new functionality introduced in this patch set is probably best done\nusing a new protocol. If it were done using the existing protocol (by\nadding a parameter in the capabilities line), we would still run into\n(a) and (c), so we might as well introduce the new protocol now.\n\nSome of the above points are repeats from my previous e-mail:\nhttps://public-inbox.org/git/20171110121347.1f7c184c543622b60164e9fb@google.com/\n\n> Some changes from that series are as follows:\n> * Lots of various cleanup on the ls-refs and fetch command code, both server\n> and client.\n> * Fetch command now supports a stateless-rpc mode which enables communicating\n> with a half-duplex connection.\n\nGood to hear about fetch support.\n\n> * Introduce a new remote-helper command 'connect-half-duplex' which is\n> implemented by remote-curl (the http remote-helper). This allows for a\n> client to establish a half-duplex connection and use remote-curl as a proxy\n> to wrap requests in http before sending them to the remote end and\n> unwrapping the responses and sending them back to the client's stdin.\n\nI'm not sure about the \"half-duplex\" name - it is half-duplex in that\neach side must terminate their communications with a flush, but not\nhalf-duplex in that request-response pairs can overlap each other (e.g.\nduring negotation during fetch - there is an optimization in which the\nclient tries to keep two requests pending at a time). I think that the\nidea we want to communicate is that requests and responses are always\npacketized, stateless, and always happen as a pair.\n\nI wonder if \"stateless-connect\" is a better keyword - it makes sense to\nme (once described) that \"stateless\" implies that the client sends\neverything the server needs at once (thus, in a packet), the server\nsends everything the client needs back at once (thus, in a packet), and\nthen the client must not assume any state-storing on the part of the\nserver or transport.\n\n> * The transport code is refactored for ls-remote, fetch, and push to provide a\n> list of ref-patterns (based on the refspec being used) when requesting refs\n> from the remote end. This allows the ls-refs code to send this list of\n> patterns so the remote end and filter the refs it sends back.\n\nBriefly looking at the implementation, the client seems to incur an\nextra roundtrip when using ls-remote (and others) with a v2-supporting\nserver. I initially didn't like this, but upon further reflection, this\nis probably fine for now. The client can be upgraded later, and I think\nthat clients will eventually want to query git-serve directly for\n\"ls-refs\" first, and then fall back to v0 for ancient servers, instead\nof checking git-upload-pack first (as in this patch set) - so, the\nsupport for \"ls-refs\" here won't be carried forward merely for backwards\ncompatibility, but will eventually be actively used.\n\nAs for the decision to use a new endpoint \"git-serve\" instead of reusing\n\"git-upload-pack\" (or \"git-receive-pack\"), reusing the existing one\nmight allow some sort of optimization later in which the first\n\"git-upload-pack\" query immediately returns with the v2 answer (instead\nof redirecting the client to \"git-serve\"), but this probably doesn't\nmatter in practice (as I stated above, I think that eventually clients\nwill query git-serve first).\n\n> This series effectively implements protocol version 2 for listing a remotes\n> refs (ls-remote) as well as for fetch for the builtin transports (ssh, git,\n> file) and for the http/https transports. Push is not implemented yet and\n> doesn't need to be implemented at the same time as fetch since the\n> receive-pack code can default to using protocol v0 when v2 is requested by the\n> client.\n\nAgreed - push can be done later.\n"},{"id":"336273","messageId":"20180109100411.b63dd903e2037372e4a28a91@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-2-bmwill@google.com","subject":"Re: [PATCH 01/26] pkt-line: introduce packet_read_with_status","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-09T18:04:11Z","receivedAt":"2018-01-09T18:04:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:03 -0800\nBrandon Williams wrote:\n\n> -int packet_read(int fd, char **src_buf, size_t *src_len,\n> -\t\tchar *buffer, unsigned size, int options)\n> +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n> +\t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n> +\t\t\t\t\t\tint options)\n> {\n> -\tint len, ret;\n> +\tint len;\n> \tchar linelen[4];\n> \n> -\tret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);\n> -\tif (ret < 0)\n> -\t\treturn ret;\n> +\tif (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)\n> +\t\treturn PACKET_READ_EOF;\n> +\n> \tlen = packet_length(linelen);\n> \tif (len < 0)\n> \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n> -\tif (!len) {\n> +\n> +\tif (len == 0) {\n\nThis change (replacing \"!len\" with \"len == 0\") is unnecessary, I think.\n\n> \t\tpacket_trace(\"0000\", 4, 0);\n> -\t\treturn 0;\n> +\t\treturn PACKET_READ_FLUSH;\n> +\t} else if (len >= 1 && len <= 3) {\n> +\t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n> \t}\n\nThis seems to be more of a \"bad line length\" than a \"bad line length\ncharacter\".\n\nAlso, some of the checks are redundant. Above, it is probably better to\ndelete \"len >= 1\", and optionally write \"len < 4\" instead of \"len <= 3\"\n(to emphasize that the subtraction of 4 below does not result in a\nnegative value).\n\n> +\n> \tlen -= 4;\n> -\tif (len >= size)\n> +\tif ((len < 0) || ((unsigned)len >= size))\n> \t\tdie(\"protocol error: bad line length %d\", len);\n\nThe \"len < 0\" check is redundant.\n\n> -\tret = get_packet_data(fd, src_buf, src_len, buffer, len, options);\n> -\tif (ret < 0)\n> -\t\treturn ret;\n> +\n> +\tif (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)\n> +\t\treturn PACKET_READ_EOF;\n> \n> \tif ((options & PACKET_READ_CHOMP_NEWLINE) &&\n> \t len && buffer[len-1] == '\\n')\n"},{"id":"336275","messageId":"20180109100813.054a6681900174ba73e30da5@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-3-bmwill@google.com","subject":"Re: [PATCH 02/26] pkt-line: introduce struct packet_reader","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-09T18:08:13Z","receivedAt":"2018-01-09T18:08:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:04 -0800\nBrandon Williams wrote:\n\n> diff --git a/pkt-line.h b/pkt-line.h\n> index 06c468927..c446e886a 100644\n> --- a/pkt-line.h\n> +++ b/pkt-line.h\n> @@ -111,6 +111,63 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n> */\n> ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);\n> \n> +struct packet_reader {\n> +\t/* source file descriptor */\n> +\tint fd;\n> +\n> +\t/* source buffer and its size */\n> +\tchar *src_buffer;\n> +\tsize_t src_len;\n> +\n> +\t/* buffer that pkt-lines are read into and its size */\n> +\tchar *buffer;\n> +\tunsigned buffer_size;\n\nIs the intention to support different buffers in the future?\n\n[snip]\n\n> +/*\n> + * Peek the next packet line without consuming it and return the status.\n> + * The next call to 'packet_reader_read()' will perform a read of the same line\n> + * that was peeked, consuming the line.\n> + *\n> + * Only a single line can be peeked at a time.\n\nIt is logical to me that if you peeked at a line, and then peeked at it\nagain, you will get the same line - I would phrase this not as a\nrestriction (\"only a single line\") but just as a statement of fact (e.g.\n\"Peeking at the same line multiple times without an intervening\npacket_reader_read will return the same result\").\n\n> + */\n> +extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);\n> +\n> #define DEFAULT_PACKET_MAX 1000\n> #define LARGE_PACKET_MAX 65520\n> #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)\n"},{"id":"336281","messageId":"20180109102735.a6d3330325aef428fde87700@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-8-bmwill@google.com","subject":"Re: [PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-09T18:27:35Z","receivedAt":"2018-01-09T18:27:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:09 -0800\nBrandon Williams wrote:\n\n> -\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> +\twhile (state != EXPECTING_DONE) {\n> +\t\tswitch (packet_reader_read(&reader)) {\n> +\t\tcase PACKET_READ_EOF:\n> +\t\t\tdie_initial_contact(1);\n> +\t\tcase PACKET_READ_NORMAL:\n> +\t\t\tlen = reader.pktlen;\n> +\t\t\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n\nThis should be a field in reader, not the global packet_buffer, I think.\n\nAlso, I did a search of usages of packet_buffer, and there are just a\nfew of them - it might be worthwhile to eliminate it, and have each\ncomponent using it allocate its own buffer. But this can be done in a\nseparate patch set.\n\n> @@ -269,6 +284,8 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t\t\tif (process_shallow(len, shallow_points))\n> \t\t\t\tbreak;\n> \t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n\nHere too.\n"},{"id":"336283","messageId":"20180109104125.4cb1a3e545cfe354862e4b8e@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-10-bmwill@google.com","subject":"Re: [PATCH 09/26] transport: store protocol version","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-09T18:41:25Z","receivedAt":"2018-01-09T18:41:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:11 -0800\nBrandon Williams wrote:\n\n> diff --git a/transport.c b/transport.c\n> index 63c3dbab9..2378dcb38 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -118,6 +118,7 @@ struct git_transport_data {\n> \tstruct child_process *conn;\n> \tint fd[2];\n> \tunsigned got_remote_heads : 1;\n> +\tenum protocol_version version;\n\nShould this be initialized to protocol_unknown_version? Right now, as\nfar as I can tell, it is zero-initialized, which means protocol_v0.\n"},{"id":"336295","messageId":"20180109190931.GA151395@google.com","threadId":"47524","inReplyTo":"20180109102735.a6d3330325aef428fde87700@google.com","subject":"Re: [PATCH 07/26] connect: convert get_remote_heads to use struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-09T19:09:31Z","receivedAt":"2018-01-09T19:09:40Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/09, Jonathan Tan wrote:\n> On Tue, 2 Jan 2018 16:18:09 -0800\n> Brandon Williams wrote:\n> \n> > -\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> > +\twhile (state != EXPECTING_DONE) {\n> > +\t\tswitch (packet_reader_read(&reader)) {\n> > +\t\tcase PACKET_READ_EOF:\n> > +\t\t\tdie_initial_contact(1);\n> > +\t\tcase PACKET_READ_NORMAL:\n> > +\t\t\tlen = reader.pktlen;\n> > +\t\t\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n> \n> This should be a field in reader, not the global packet_buffer, I think.\n\nThanks for catching that.\n\n> \n> Also, I did a search of usages of packet_buffer, and there are just a\n> few of them - it might be worthwhile to eliminate it, and have each\n> component using it allocate its own buffer. But this can be done in a\n> separate patch set.\n\nI'll go through and eliminate the references to packet_buffer by passing\nin the buffer explicitly.\n\n> \n> > @@ -269,6 +284,8 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > \t\t\tif (process_shallow(len, shallow_points))\n> > \t\t\t\tbreak;\n> > \t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n> \n> Here too.\n\n-- \nBrandon Williams\n"},{"id":"336298","messageId":"20180109191510.GB151395@google.com","threadId":"47524","inReplyTo":"20180109104125.4cb1a3e545cfe354862e4b8e@google.com","subject":"Re: [PATCH 09/26] transport: store protocol version","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-09T19:15:10Z","receivedAt":"2018-01-09T19:15:17Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/09, Jonathan Tan wrote:\n> On Tue, 2 Jan 2018 16:18:11 -0800\n> Brandon Williams wrote:\n> \n> > diff --git a/transport.c b/transport.c\n> > index 63c3dbab9..2378dcb38 100644\n> > --- a/transport.c\n> > +++ b/transport.c\n> > @@ -118,6 +118,7 @@ struct git_transport_data {\n> > \tstruct child_process *conn;\n> > \tint fd[2];\n> > \tunsigned got_remote_heads : 1;\n> > +\tenum protocol_version version;\n> \n> Should this be initialized to protocol_unknown_version? Right now, as\n> far as I can tell, it is zero-initialized, which means protocol_v0.\n\nI don't think it matters as the value isn't used until after the\nversion has already been discovered.\n\n-- \nBrandon Williams\n"},{"id":"336299","messageId":"20180109191904.GC151395@google.com","threadId":"47524","inReplyTo":"20180109100813.054a6681900174ba73e30da5@google.com","subject":"Re: [PATCH 02/26] pkt-line: introduce struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-09T19:19:04Z","receivedAt":"2018-01-09T19:19:11Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/09, Jonathan Tan wrote:\n> On Tue, 2 Jan 2018 16:18:04 -0800\n> Brandon Williams wrote:\n> \n> > diff --git a/pkt-line.h b/pkt-line.h\n> > index 06c468927..c446e886a 100644\n> > --- a/pkt-line.h\n> > +++ b/pkt-line.h\n> > @@ -111,6 +111,63 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n> > */\n> > ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);\n> > \n> > +struct packet_reader {\n> > +\t/* source file descriptor */\n> > +\tint fd;\n> > +\n> > +\t/* source buffer and its size */\n> > +\tchar *src_buffer;\n> > +\tsize_t src_len;\n> > +\n> > +\t/* buffer that pkt-lines are read into and its size */\n> > +\tchar *buffer;\n> > +\tunsigned buffer_size;\n> \n> Is the intention to support different buffers in the future?\n\nPotentially at some point.\n\n> \n> [snip]\n> \n> > +/*\n> > + * Peek the next packet line without consuming it and return the status.\n> > + * The next call to 'packet_reader_read()' will perform a read of the same line\n> > + * that was peeked, consuming the line.\n> > + *\n> > + * Only a single line can be peeked at a time.\n> \n> It is logical to me that if you peeked at a line, and then peeked at it\n> again, you will get the same line - I would phrase this not as a\n> restriction (\"only a single line\") but just as a statement of fact (e.g.\n> \"Peeking at the same line multiple times without an intervening\n> packet_reader_read will return the same result\").\n\nFair enough, i'll change the wording.\n\n> \n> > + */\n> > +extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);\n> > +\n> > #define DEFAULT_PACKET_MAX 1000\n> > #define LARGE_PACKET_MAX 65520\n> > #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)\n\n-- \nBrandon Williams\n"},{"id":"336302","messageId":"20180109192801.GD151395@google.com","threadId":"47524","inReplyTo":"20180109100411.b63dd903e2037372e4a28a91@google.com","subject":"Re: [PATCH 01/26] pkt-line: introduce packet_read_with_status","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-09T19:28:01Z","receivedAt":"2018-01-09T19:28:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/09, Jonathan Tan wrote:\n> On Tue, 2 Jan 2018 16:18:03 -0800\n> Brandon Williams wrote:\n> \n> > -int packet_read(int fd, char **src_buf, size_t *src_len,\n> > -\t\tchar *buffer, unsigned size, int options)\n> > +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n> > +\t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n> > +\t\t\t\t\t\tint options)\n> > {\n> > -\tint len, ret;\n> > +\tint len;\n> > \tchar linelen[4];\n> > \n> > -\tret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);\n> > -\tif (ret < 0)\n> > -\t\treturn ret;\n> > +\tif (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)\n> > +\t\treturn PACKET_READ_EOF;\n> > +\n> > \tlen = packet_length(linelen);\n> > \tif (len < 0)\n> > \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n> > -\tif (!len) {\n> > +\n> > +\tif (len == 0) {\n> \n> This change (replacing \"!len\" with \"len == 0\") is unnecessary, I think.\n> \n> > \t\tpacket_trace(\"0000\", 4, 0);\n> > -\t\treturn 0;\n> > +\t\treturn PACKET_READ_FLUSH;\n> > +\t} else if (len >= 1 && len <= 3) {\n> > +\t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n> > \t}\n> \n> This seems to be more of a \"bad line length\" than a \"bad line length\n> character\".\n\nI'll make these changes, though I do think this needs to stay as a \"bad\nline length character\" as the len could be neg which is an indication of\nparsing the linelen character failed.\n\n> \n> Also, some of the checks are redundant. Above, it is probably better to\n> delete \"len >= 1\", and optionally write \"len < 4\" instead of \"len <= 3\"\n> (to emphasize that the subtraction of 4 below does not result in a\n> negative value).\n> \n> > +\n> > \tlen -= 4;\n> > -\tif (len >= size)\n> > +\tif ((len < 0) || ((unsigned)len >= size))\n> > \t\tdie(\"protocol error: bad line length %d\", len);\n> \n> The \"len < 0\" check is redundant.\n> \n> > -\tret = get_packet_data(fd, src_buf, src_len, buffer, len, options);\n> > -\tif (ret < 0)\n> > -\t\treturn ret;\n> > +\n> > +\tif (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)\n> > +\t\treturn PACKET_READ_EOF;\n> > \n> > \tif ((options & PACKET_READ_CHOMP_NEWLINE) &&\n> > \t len && buffer[len-1] == '\\n')\n\n-- \nBrandon Williams\n"},{"id":"336313","messageId":"20180109122455.5845c4f12ccdd4cada1528f2@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-12-bmwill@google.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-09T20:24:55Z","receivedAt":"2018-01-09T20:25:03Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:13 -0800\nBrandon Williams wrote:\n\n> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> new file mode 100644\n> index 000000000..b87ba3816\n> --- /dev/null\n> +++ b/Documentation/technical/protocol-v2.txt\n\nI'll review the documentation later, once there is some consensus that\nthe overall design is OK. (Or maybe there already is consensus?)\n\n> diff --git a/builtin/serve.c b/builtin/serve.c\n> new file mode 100644\n> index 000000000..bb726786a\n> --- /dev/null\n> +++ b/builtin/serve.c\n> @@ -0,0 +1,30 @@\n> +#include \"cache.h\"\n> +#include \"builtin.h\"\n> +#include \"parse-options.h\"\n> +#include \"serve.h\"\n> +\n> +static char const * const grep_usage[] = {\n\nShould be serve_usage.\n\n> diff --git a/serve.c b/serve.c\n> new file mode 100644\n> index 000000000..da8127775\n> --- /dev/null\n> +++ b/serve.c\n\n[snip]\n\n> +struct protocol_capability {\n> +\tconst char *name; /* capability name */\n\nMaybe document as:\n\n The name of the capability. The server uses this name when advertising\n this capability, and the client uses this name to invoke the command\n corresponding to this capability.\n\n> +\t/*\n> +\t * Function queried to see if a capability should be advertised.\n> +\t * Optionally a value can be specified by adding it to 'value'.\n> +\t */\n> +\tint (*advertise)(struct repository *r, struct strbuf *value);\n\nDocument what happens when value is appended to. For example:\n\n ... If value is appended to, the server will advertise this capability\n as = instead of .\n\n> +\t/*\n> +\t * Function called when a client requests the capability as a command.\n> +\t * The command request will be provided to the function via 'keys', the\n> +\t * capabilities requested, and 'args', the command specific parameters.\n> +\t *\n> +\t * This field should be NULL for capabilities which are not commands.\n> +\t */\n> +\tint (*command)(struct repository *r,\n> +\t\t struct argv_array *keys,\n> +\t\t struct argv_array *args);\n\nLooking at the code below, I see that the command is not executed unless\nadvertise returns true - this means that a command cannot be both\nsupported and unadvertised. Would this be too restrictive? For example,\nthis would disallow a gradual across-multiple-servers rollout in which\nwe allow but not advertise a capability, and then after some time,\nadvertise the capability.\n\nIf we change this, then the value parameter of advertise can be\nmandatory instead of optional.\n"},{"id":"336316","messageId":"20180109125025.6c88700e6e00e45822ee8932@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-13-bmwill@google.com","subject":"Re: [PATCH 12/26] ls-refs: introduce ls-refs server command","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-09T20:50:25Z","receivedAt":"2018-01-09T20:50:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:14 -0800\nBrandon Williams wrote:\n\n> + symrefs: In addition to the object pointed by it, show the underlying\n> +\t ref pointed by it when showing a symbolic ref.\n> + peel: Show peeled tags.\n> + ref-pattern : When specified, only references matching the\n> +\t\t\t given patterns are displayed.\n\nI notice \"symrefs\" being tested in patch 13 and \"ref-pattern\" being\ntested in patch 16. Is it possible to make a test for \"peel\" as well?\n(Or is it being tested somewhere I didn't notice?)\n"},{"id":"336323","messageId":"20180109221642.GF151395@google.com","threadId":"47524","inReplyTo":"20180109122455.5845c4f12ccdd4cada1528f2@google.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-09T22:16:42Z","receivedAt":"2018-01-09T22:16:50Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/09, Jonathan Tan wrote:\n> On Tue, 2 Jan 2018 16:18:13 -0800\n> Brandon Williams wrote:\n> \n> > diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> > new file mode 100644\n> > index 000000000..b87ba3816\n> > --- /dev/null\n> > +++ b/Documentation/technical/protocol-v2.txt\n> \n> I'll review the documentation later, once there is some consensus that\n> the overall design is OK. (Or maybe there already is consensus?)\n> \n> > diff --git a/builtin/serve.c b/builtin/serve.c\n> > new file mode 100644\n> > index 000000000..bb726786a\n> > --- /dev/null\n> > +++ b/builtin/serve.c\n> > @@ -0,0 +1,30 @@\n> > +#include \"cache.h\"\n> > +#include \"builtin.h\"\n> > +#include \"parse-options.h\"\n> > +#include \"serve.h\"\n> > +\n> > +static char const * const grep_usage[] = {\n> \n> Should be serve_usage.\n> \n> > diff --git a/serve.c b/serve.c\n> > new file mode 100644\n> > index 000000000..da8127775\n> > --- /dev/null\n> > +++ b/serve.c\n> \n> [snip]\n> \n> > +struct protocol_capability {\n> > +\tconst char *name; /* capability name */\n> \n> Maybe document as:\n> \n> The name of the capability. The server uses this name when advertising\n> this capability, and the client uses this name to invoke the command\n> corresponding to this capability.\n> \n> > +\t/*\n> > +\t * Function queried to see if a capability should be advertised.\n> > +\t * Optionally a value can be specified by adding it to 'value'.\n> > +\t */\n> > +\tint (*advertise)(struct repository *r, struct strbuf *value);\n> \n> Document what happens when value is appended to. For example:\n> \n> ... If value is appended to, the server will advertise this capability\n> as = instead of .\n> \n\nAll good documentation changes.\n\n> > +\t/*\n> > +\t * Function called when a client requests the capability as a command.\n> > +\t * The command request will be provided to the function via 'keys', the\n> > +\t * capabilities requested, and 'args', the command specific parameters.\n> > +\t *\n> > +\t * This field should be NULL for capabilities which are not commands.\n> > +\t */\n> > +\tint (*command)(struct repository *r,\n> > +\t\t struct argv_array *keys,\n> > +\t\t struct argv_array *args);\n> \n> Looking at the code below, I see that the command is not executed unless\n> advertise returns true - this means that a command cannot be both\n> supported and unadvertised. Would this be too restrictive? For example,\n> this would disallow a gradual across-multiple-servers rollout in which\n> we allow but not advertise a capability, and then after some time,\n> advertise the capability.\n\nOne way to change this would be to just add another function to the\nstruct which is called to check if the command is allowed, instead of\nrelying on the same function to do that for both advertise and\nallow...though I don't see a big win for allowing a command but not\nadvertising it.\n\n> \n> If we change this, then the value parameter of advertise can be\n> mandatory instead of optional.\n\nI don't see how this fixes the issue you bring up.\n\n-- \nBrandon Williams\n"},{"id":"336324","messageId":"20180109142451.1e10f3936eafc270c9bb8b76@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-14-bmwill@google.com","subject":"Re: [PATCH 13/26] connect: request remote refs using v2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-09T22:24:51Z","receivedAt":"2018-01-09T22:24:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:15 -0800\nBrandon Williams wrote:\n\n> diff --git a/connect.c b/connect.c\n> index caa539b75..9badd403f 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -12,9 +12,11 @@\n> #include \"sha1-array.h\"\n> #include \"transport.h\"\n> #include \"strbuf.h\"\n> +#include \"version.h\"\n> #include \"protocol.h\"\n> \n> static char *server_capabilities;\n> +static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n> static const char *parse_feature_value(const char *, const char *, int *);\n> \n> static int check_ref(const char *name, unsigned int flags)\n> @@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)\n> \t\t \"and the repository exists.\"));\n> }\n> \n> +static int server_supports_v2(const char *c, int die_on_error)\n\nDocument what \"c\" means.\n\n[snip]\n\n> +static void process_capabilities_v2(struct packet_reader *reader)\n> +{\n> +\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n> +\t\targv_array_push(&server_capabilities_v2, reader->line);\n> +\t}\n\nNo need for braces on single-line blocks.\n\n> +static int process_ref_v2(const char *line, struct ref ***list)\n\nThe \"list\" is the tail of a linked list, so maybe name it \"tail\"\ninstead.\n"},{"id":"336325","messageId":"20180109142827.a2fe13a69e5240b6a12729e1@google.com","threadId":"47524","inReplyTo":"20180109221642.GF151395@google.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-09T22:28:27Z","receivedAt":"2018-01-09T22:28:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 9 Jan 2018 14:16:42 -0800\nBrandon Williams wrote:\n\n> All good documentation changes.\n\nThanks!\n\n> > > +\t/*\n> > > +\t * Function called when a client requests the capability as a command.\n> > > +\t * The command request will be provided to the function via 'keys', the\n> > > +\t * capabilities requested, and 'args', the command specific parameters.\n> > > +\t *\n> > > +\t * This field should be NULL for capabilities which are not commands.\n> > > +\t */\n> > > +\tint (*command)(struct repository *r,\n> > > +\t\t struct argv_array *keys,\n> > > +\t\t struct argv_array *args);\n> > \n> > Looking at the code below, I see that the command is not executed unless\n> > advertise returns true - this means that a command cannot be both\n> > supported and unadvertised. Would this be too restrictive? For example,\n> > this would disallow a gradual across-multiple-servers rollout in which\n> > we allow but not advertise a capability, and then after some time,\n> > advertise the capability.\n> \n> One way to change this would be to just add another function to the\n> struct which is called to check if the command is allowed, instead of\n> relying on the same function to do that for both advertise and\n> allow...though I don't see a big win for allowing a command but not\n> advertising it.\n\nMy rationale for allowing a command but not advertising it is in the\nparagraph above (that you quoted), but if that is insufficient\nrationale, then I agree that we don't need to do this.\n\n> > If we change this, then the value parameter of advertise can be\n> > mandatory instead of optional.\n> \n> I don't see how this fixes the issue you bring up.\n\nThis is a consequence, not a fix - if we were to do as I suggested, then\nwe no longer need to invoke advertise to check whether something is\nadvertised except when we are advertising them, in which case \"value\"\nnever needs to be NULL.\n"},{"id":"336326","messageId":"20180109223455.GG151395@google.com","threadId":"47524","inReplyTo":"20180109142827.a2fe13a69e5240b6a12729e1@google.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-09T22:34:55Z","receivedAt":"2018-01-09T22:35:03Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/09, Jonathan Tan wrote:\n> On Tue, 9 Jan 2018 14:16:42 -0800\n> Brandon Williams wrote:\n> \n> > All good documentation changes.\n> \n> Thanks!\n> \n> > > > +\t/*\n> > > > +\t * Function called when a client requests the capability as a command.\n> > > > +\t * The command request will be provided to the function via 'keys', the\n> > > > +\t * capabilities requested, and 'args', the command specific parameters.\n> > > > +\t *\n> > > > +\t * This field should be NULL for capabilities which are not commands.\n> > > > +\t */\n> > > > +\tint (*command)(struct repository *r,\n> > > > +\t\t struct argv_array *keys,\n> > > > +\t\t struct argv_array *args);\n> > > \n> > > Looking at the code below, I see that the command is not executed unless\n> > > advertise returns true - this means that a command cannot be both\n> > > supported and unadvertised. Would this be too restrictive? For example,\n> > > this would disallow a gradual across-multiple-servers rollout in which\n> > > we allow but not advertise a capability, and then after some time,\n> > > advertise the capability.\n> > \n> > One way to change this would be to just add another function to the\n> > struct which is called to check if the command is allowed, instead of\n> > relying on the same function to do that for both advertise and\n> > allow...though I don't see a big win for allowing a command but not\n> > advertising it.\n> \n> My rationale for allowing a command but not advertising it is in the\n> paragraph above (that you quoted), but if that is insufficient\n> rationale, then I agree that we don't need to do this.\n\nI have no issues with adding that functionality, i don't really feel\nthat strongly one way or another. Just seemed like additional work for\nnot much gain right now, key being right now. It very well may be worth\nit for the use case you specified. If so I can definitely make the\nchange.\n\n> \n> > > If we change this, then the value parameter of advertise can be\n> > > mandatory instead of optional.\n> > \n> > I don't see how this fixes the issue you bring up.\n> \n> This is a consequence, not a fix - if we were to do as I suggested, then\n> we no longer need to invoke advertise to check whether something is\n> advertised except when we are advertising them, in which case \"value\"\n> never needs to be NULL.\n\nOh I understand what you are trying to explain, yes you're right.\n\n-- \nBrandon Williams\n"},{"id":"336332","messageId":"20180109160512.b3f12f45afd7121d25cf5fa4@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-21-bmwill@google.com","subject":"Re: [PATCH 20/26] fetch-pack: perform a fetch using v2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-10T00:05:12Z","receivedAt":"2018-01-10T00:05:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:22 -0800\nBrandon Williams wrote:\n\n> +static enum ack_type process_ack(const char *line, struct object_id *oid)\n> +{\n> +\tconst char *arg;\n> +\n> +\tif (!strcmp(line, \"NAK\"))\n> +\t\treturn NAK;\n> +\tif (skip_prefix(line, \"ACK \", &arg)) {\n> +\t\tif (!parse_oid_hex(arg, oid, &arg)) {\n> +\t\t\tif (strstr(arg, \"continue\"))\n> +\t\t\t\treturn ACK_continue;\n\nThis function seems to be only used for v2, so I don't think we need to\nparse \"continue\".\n\nAlso, maybe describe the plan for supporting functionality not supported\nyet (e.g. server-side declaration of shallows and client-side \"deepen\").\n\nIt may be possible to delay support for server-side shallows on the\nserver (that is, only implement support for it in the client) since the\nserver can just declare that it doesn't support protocol v2 when serving\nsuch repos (although it might just be easier to implement server-side\nsupport in this case).\n\nFor \"deepen\", we need support for it both on the client and the server\nnow unless we plan to declare a \"deepen\" capability in the future (then,\nas of these patches, clients that require \"deepen\" will use protocol v1;\nwhen a new server declares \"deepen\", old clients will ignore it and keep\nthe status quo, and new clients can then use \"deepen\").\n\nThere may be others that I've missed.\n"},{"id":"336333","messageId":"20180109161030.e04f0a8d21eb1bdb0f626a83@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-27-bmwill@google.com","subject":"Re: [PATCH 26/26] remote-curl: implement connect-half-duplex command","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-10T00:10:30Z","receivedAt":"2018-01-10T00:10:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:28 -0800\nBrandon Williams wrote:\n\n> Teach remote-curl the 'connect-half-duplex' command which is used to\n> establish a half-duplex connection with servers which support protocol\n> version 2. This allows remote-curl to act as a proxy, allowing the git\n> client to communicate natively with a remote end, simply using\n> remote-curl as a pass through to convert requests to http.\n> \n> Signed-off-by: Brandon Williams \n> ---\n> remote-curl.c | 185 ++++++++++++++++++++++++++++++++++++++++++++++++-\n> t/t5701-protocol-v2.sh | 41 +++++++++++\n> 2 files changed, 224 insertions(+), 2 deletions(-)\n\nI didn't look at the usage of the curl API in detail, but overall this\nlooks good. I'm pleasantly surprised that it didn't take so many lines\nof code as I expected.\n\nOverall everything looks good, except for the points that I have brought\nup in my other e-mails.\n\n> diff --git a/remote-curl.c b/remote-curl.c\n> index 4086aa733..b63b06398 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n\n[snip]\n\n> +struct proxy_state {\n> +\tchar *service_name;\n> +\tchar *service_url;\n> +\tchar *hdr_content_type;\n> +\tchar *hdr_accept;\n\nMaybe document that the above 3 fields (service_url to hdr_accept) are\ncached because we need to pass them to curl_easy_setopt() for every\nrequest.\n"},{"id":"336400","messageId":"20180110095730.1daaa70d7c583c58b85142a3@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-27-bmwill@google.com","subject":"Re: [PATCH 26/26] remote-curl: implement connect-half-duplex command","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-10T17:57:30Z","receivedAt":"2018-01-10T17:57:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 2 Jan 2018 16:18:28 -0800\nBrandon Williams wrote:\n\n> +static size_t proxy_in(void *ptr, size_t eltsize,\n> +\t\t size_t nmemb, void *buffer_)\n\nOK, I managed to look at the Curl stuff in more detail.\n\nI know that these parameter names are what remote_curl.c has been using\nfor its callbacks, but I find them confusing (in particular, some Curl\ndocumentation rightly refer to the 1st parameter as a buffer, and the\n4th parameter is actually userdata). Also, according to the Curl\ndocumentation, the type of the first parameter is \"char *\". Could we\nchange the type of the first parameter to \"char *\", and the name of the\nfourth parameter either to \"proxy_state_\" or \"userdata\"?\n\n> +{\n> +\tsize_t max = eltsize * nmemb;\n> +\tstruct proxy_state *p = buffer_;\n> +\tsize_t avail = p->request_buffer.len - p->pos;\n> +\n> +\tif (!avail) {\n> +\t\tif (p->seen_flush) {\n> +\t\t\tp->seen_flush = 0;\n> +\t\t\treturn 0;\n> +\t\t}\n> +\n> +\t\tstrbuf_reset(&p->request_buffer);\n> +\t\tswitch (packet_reader_read(&p->reader)) {\n> +\t\tcase PACKET_READ_EOF:\n> +\t\t\tdie(\"error reading request from parent process\");\n\nThis should say \"BUG:\", I think. I'm not sure what the best way of\nexplaining it is, but basically connect_half_duplex is supposed to\nensure (by peeking) that there is no EOF when proxy_in() is called.\n\n> +\t\tcase PACKET_READ_NORMAL:\n> +\t\t\tpacket_buf_write_len(&p->request_buffer, p->reader.line,\n> +\t\t\t\t\t p->reader.pktlen);\n> +\t\t\tbreak;\n> +\t\tcase PACKET_READ_DELIM:\n> +\t\t\tpacket_buf_delim(&p->request_buffer);\n> +\t\t\tbreak;\n> +\t\tcase PACKET_READ_FLUSH:\n> +\t\t\tpacket_buf_flush(&p->request_buffer);\n> +\t\t\tp->seen_flush = 1;\n> +\t\t\tbreak;\n> +\t\t}\n> +\t\tp->pos = 0;\n> +\t\tavail = p->request_buffer.len;\n> +\t}\n> +\n> +\tif (max < avail)\n> +\t\tavail = max;\n> +\tmemcpy(ptr, p->request_buffer.buf + p->pos, avail);\n> +\tp->pos += avail;\n> +\treturn avail;\n\nThanks, this looks correct. I wish that the Curl API had a way for us to\nsay \"here are 4 more bytes, and that is all\" instead of us having to\nmake a note (p->seen_flush) to remember to return 0 on the next call,\nbut that's the way it is.\n\n> +}\n> +static size_t proxy_out(char *ptr, size_t eltsize,\n> +\t\t\tsize_t nmemb, void *buffer_)\n\nAdd a blank line before proxy_out. Also, same comment as proxy_in()\nabout the function signature.\n\n> +{\n> +\tsize_t size = eltsize * nmemb;\n> +\tstruct proxy_state *p = buffer_;\n> +\n> +\twrite_or_die(p->out, ptr, size);\n> +\treturn size;\n> +}\n> +\n> +static int proxy_post(struct proxy_state *p)\n> +{\n> +\tstruct active_request_slot *slot;\n> +\tstruct curl_slist *headers = http_copy_default_headers();\n> +\tint err;\n> +\n> +\theaders = curl_slist_append(headers, p->hdr_content_type);\n> +\theaders = curl_slist_append(headers, p->hdr_accept);\n> +\theaders = curl_slist_append(headers, \"Transfer-Encoding: chunked\");\n> +\n> +\tslot = get_active_slot();\n> +\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n\nI looked at the Curl documentation for CURLOPT_HTTPHEADER and\ncurl_easy_setopt doesn't consume the argument here (in fact, it asks us\nto keep \"headers\" around), so it might be possible to just generate the\nheaders once in proxy_state_init().\n\n> +\n> +\t/* Setup function to read request from client */\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_READDATA, p);\n> +\n> +\t/* Setup function to write server response to client */\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);\n> +\n> +\terr = run_slot(slot, NULL);\n> +\n> +\tif (err != HTTP_OK)\n> +\t\terr = -1;\n\nThis seems to mean that we cannot have two requests in flight at the\nsame time even while there is no response (from the fact that we have a\nHTTP status code after returning from run_slot()).\n\nI thought that git fetch over HTTP uses the two-requests-in-flight\noptimization that it also does over other protocols like SSH, but I see\nthat that code path (fetch_git() in remote-curl.c) also uses run_slot()\nindirectly, so maybe my assumption is wrong. Anyway, this is outside the\nscope of this patch.\n\n> +\n> +\tcurl_slist_free_all(headers);\n> +\treturn err;\n> +}\n> +\n> +static int connect_half_duplex(const char *service_name)\n> +{\n> +\tstruct discovery *discover;\n> +\tstruct proxy_state p;\n> +\n> +\t/*\n> +\t * Run the info/refs request and see if the server supports protocol\n> +\t * v2. If and only if the server supports v2 can we successfully\n> +\t * establish a half-duplex connection, otherwise we need to tell the\n> +\t * client to fallback to using other transport helper functions to\n> +\t * complete their request.\n> +\t */\n> +\tdiscover = discover_refs(service_name, 0);\n> +\tif (discover->version != protocol_v2) {\n> +\t\tprintf(\"fallback\\n\");\n> +\t\tfflush(stdout);\n> +\t\treturn -1;\n> +\t} else {\n> +\t\t/* Half-Duplex Connection established */\n> +\t\tprintf(\"\\n\");\n> +\t\tfflush(stdout);\n> +\t}\n> +\n> +\tproxy_state_init(&p, service_name);\n> +\n> +\t/*\n> +\t * Dump the capability listing that we got from the server earlier\n> +\t * during the info/refs request.\n> +\t */\n> +\twrite_or_die(p.out, discover->buf, discover->len);\n> +\n> +\t/* Peek the next packet line. Until we see EOF keep sending POSTs */\n> +\twhile (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {\n> +\t\tif (proxy_post(&p)) {\n> +\t\t\t/* We would have an err here */\n\nProbably better to comment \"Error message already printed by\nproxy_post\".\n"},{"id":"336437","messageId":"20180111002355.GA91567@google.com","threadId":"47524","inReplyTo":"20180109095534.c0c9b9ad3933d406c993c3ab@google.com","subject":"Re: [PATCH 00/26] protocol version 2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-11T00:23:55Z","receivedAt":"2018-01-11T00:24:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/09, Jonathan Tan wrote:\n> On Tue, 2 Jan 2018 16:18:02 -0800\n> Brandon Williams wrote:\n> \n> > * Introduce a new remote-helper command 'connect-half-duplex' which is\n> > implemented by remote-curl (the http remote-helper). This allows for a\n> > client to establish a half-duplex connection and use remote-curl as a proxy\n> > to wrap requests in http before sending them to the remote end and\n> > unwrapping the responses and sending them back to the client's stdin.\n> \n> I'm not sure about the \"half-duplex\" name - it is half-duplex in that\n> each side must terminate their communications with a flush, but not\n> half-duplex in that request-response pairs can overlap each other (e.g.\n> during negotation during fetch - there is an optimization in which the\n> client tries to keep two requests pending at a time). I think that the\n> idea we want to communicate is that requests and responses are always\n> packetized, stateless, and always happen as a pair.\n> \n> I wonder if \"stateless-connect\" is a better keyword - it makes sense to\n> me (once described) that \"stateless\" implies that the client sends\n> everything the server needs at once (thus, in a packet), the server\n> sends everything the client needs back at once (thus, in a packet), and\n> then the client must not assume any state-storing on the part of the\n> server or transport.\n\nI like that name much better, I think I'll change it to use\n'stateless-connect'. Thanks :)\n\n\n-- \nBrandon Williams\n"},{"id":"336439","messageId":"20180111010920.GB91567@google.com","threadId":"47524","inReplyTo":"20180110095730.1daaa70d7c583c58b85142a3@google.com","subject":"Re: [PATCH 26/26] remote-curl: implement connect-half-duplex command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-11T01:09:20Z","receivedAt":"2018-01-11T01:09:29Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/10, Jonathan Tan wrote:\n> On Tue, 2 Jan 2018 16:18:28 -0800\n> Brandon Williams wrote:\n> \n> > +static size_t proxy_in(void *ptr, size_t eltsize,\n> > +\t\t size_t nmemb, void *buffer_)\n> \n> OK, I managed to look at the Curl stuff in more detail.\n> \n> I know that these parameter names are what remote_curl.c has been using\n> for its callbacks, but I find them confusing (in particular, some Curl\n> documentation rightly refer to the 1st parameter as a buffer, and the\n> 4th parameter is actually userdata). Also, according to the Curl\n> documentation, the type of the first parameter is \"char *\". Could we\n> change the type of the first parameter to \"char *\", and the name of the\n> fourth parameter either to \"proxy_state_\" or \"userdata\"?\n\nSounds good, I'll make the change.\n\n> \n> > +{\n> > +\tsize_t max = eltsize * nmemb;\n> > +\tstruct proxy_state *p = buffer_;\n> > +\tsize_t avail = p->request_buffer.len - p->pos;\n> > +\n> > +\tif (!avail) {\n> > +\t\tif (p->seen_flush) {\n> > +\t\t\tp->seen_flush = 0;\n> > +\t\t\treturn 0;\n> > +\t\t}\n> > +\n> > +\t\tstrbuf_reset(&p->request_buffer);\n> > +\t\tswitch (packet_reader_read(&p->reader)) {\n> > +\t\tcase PACKET_READ_EOF:\n> > +\t\t\tdie(\"error reading request from parent process\");\n> \n> This should say \"BUG:\", I think. I'm not sure what the best way of\n> explaining it is, but basically connect_half_duplex is supposed to\n> ensure (by peeking) that there is no EOF when proxy_in() is called.\n\nThis wouldn't necessarily be a bug if the parent dies early for some\nreason though right?\n\n> \n> > +\t\tcase PACKET_READ_NORMAL:\n> > +\t\t\tpacket_buf_write_len(&p->request_buffer, p->reader.line,\n> > +\t\t\t\t\t p->reader.pktlen);\n> > +\t\t\tbreak;\n> > +\t\tcase PACKET_READ_DELIM:\n> > +\t\t\tpacket_buf_delim(&p->request_buffer);\n> > +\t\t\tbreak;\n> > +\t\tcase PACKET_READ_FLUSH:\n> > +\t\t\tpacket_buf_flush(&p->request_buffer);\n> > +\t\t\tp->seen_flush = 1;\n> > +\t\t\tbreak;\n> > +\t\t}\n> > +\t\tp->pos = 0;\n> > +\t\tavail = p->request_buffer.len;\n> > +\t}\n> > +\n> > +\tif (max < avail)\n> > +\t\tavail = max;\n> > +\tmemcpy(ptr, p->request_buffer.buf + p->pos, avail);\n> > +\tp->pos += avail;\n> > +\treturn avail;\n> \n> Thanks, this looks correct. I wish that the Curl API had a way for us to\n> say \"here are 4 more bytes, and that is all\" instead of us having to\n> make a note (p->seen_flush) to remember to return 0 on the next call,\n> but that's the way it is.\n> \n> > +}\n> > +static size_t proxy_out(char *ptr, size_t eltsize,\n> > +\t\t\tsize_t nmemb, void *buffer_)\n> \n> Add a blank line before proxy_out. Also, same comment as proxy_in()\n> about the function signature.\n\nI'll change this function too.\n\n> \n> > +{\n> > +\tsize_t size = eltsize * nmemb;\n> > +\tstruct proxy_state *p = buffer_;\n> > +\n> > +\twrite_or_die(p->out, ptr, size);\n> > +\treturn size;\n> > +}\n> > +\n> > +static int proxy_post(struct proxy_state *p)\n> > +{\n> > +\tstruct active_request_slot *slot;\n> > +\tstruct curl_slist *headers = http_copy_default_headers();\n> > +\tint err;\n> > +\n> > +\theaders = curl_slist_append(headers, p->hdr_content_type);\n> > +\theaders = curl_slist_append(headers, p->hdr_accept);\n> > +\theaders = curl_slist_append(headers, \"Transfer-Encoding: chunked\");\n> > +\n> > +\tslot = get_active_slot();\n> > +\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n> \n> I looked at the Curl documentation for CURLOPT_HTTPHEADER and\n> curl_easy_setopt doesn't consume the argument here (in fact, it asks us\n> to keep \"headers\" around), so it might be possible to just generate the\n> headers once in proxy_state_init().\n\nYeah I'll go ahead and do that, it'll make the post function a bit\ncleaner too.\n\n> \n> > +\n> > +\t/* Setup function to read request from client */\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_READDATA, p);\n> > +\n> > +\t/* Setup function to write server response to client */\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);\n> > +\n> > +\terr = run_slot(slot, NULL);\n> > +\n> > +\tif (err != HTTP_OK)\n> > +\t\terr = -1;\n> \n> This seems to mean that we cannot have two requests in flight at the\n> same time even while there is no response (from the fact that we have a\n> HTTP status code after returning from run_slot()).\n> \n> I thought that git fetch over HTTP uses the two-requests-in-flight\n> optimization that it also does over other protocols like SSH, but I see\n> that that code path (fetch_git() in remote-curl.c) also uses run_slot()\n> indirectly, so maybe my assumption is wrong. Anyway, this is outside the\n> scope of this patch.\n> \n> > +\n> > +\tcurl_slist_free_all(headers);\n> > +\treturn err;\n> > +}\n> > +\n> > +static int connect_half_duplex(const char *service_name)\n> > +{\n> > +\tstruct discovery *discover;\n> > +\tstruct proxy_state p;\n> > +\n> > +\t/*\n> > +\t * Run the info/refs request and see if the server supports protocol\n> > +\t * v2. If and only if the server supports v2 can we successfully\n> > +\t * establish a half-duplex connection, otherwise we need to tell the\n> > +\t * client to fallback to using other transport helper functions to\n> > +\t * complete their request.\n> > +\t */\n> > +\tdiscover = discover_refs(service_name, 0);\n> > +\tif (discover->version != protocol_v2) {\n> > +\t\tprintf(\"fallback\\n\");\n> > +\t\tfflush(stdout);\n> > +\t\treturn -1;\n> > +\t} else {\n> > +\t\t/* Half-Duplex Connection established */\n> > +\t\tprintf(\"\\n\");\n> > +\t\tfflush(stdout);\n> > +\t}\n> > +\n> > +\tproxy_state_init(&p, service_name);\n> > +\n> > +\t/*\n> > +\t * Dump the capability listing that we got from the server earlier\n> > +\t * during the info/refs request.\n> > +\t */\n> > +\twrite_or_die(p.out, discover->buf, discover->len);\n> > +\n> > +\t/* Peek the next packet line. Until we see EOF keep sending POSTs */\n> > +\twhile (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {\n> > +\t\tif (proxy_post(&p)) {\n> > +\t\t\t/* We would have an err here */\n> \n> Probably better to comment \"Error message already printed by\n> proxy_post\".\n\n-- \nBrandon Williams\n"},{"id":"336670","messageId":"20180116192303.GC135179@google.com","threadId":"47524","inReplyTo":"20180109125025.6c88700e6e00e45822ee8932@google.com","subject":"Re: [PATCH 12/26] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-16T19:23:03Z","receivedAt":"2018-01-16T19:23:11Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/09, Jonathan Tan wrote:\n> On Tue, 2 Jan 2018 16:18:14 -0800\n> Brandon Williams wrote:\n> \n> > + symrefs: In addition to the object pointed by it, show the underlying\n> > +\t ref pointed by it when showing a symbolic ref.\n> > + peel: Show peeled tags.\n> > + ref-pattern : When specified, only references matching the\n> > +\t\t\t given patterns are displayed.\n> \n> I notice \"symrefs\" being tested in patch 13 and \"ref-pattern\" being\n> tested in patch 16. Is it possible to make a test for \"peel\" as well?\n> (Or is it being tested somewhere I didn't notice?)\n\nReally good suggestion. I'll introduce unit tests for both the\ngit-serve cmdline as well as for more simple server commands (ls-refs).\n\n-- \nBrandon Williams\n"},{"id":"337421","messageId":"20180125235838.138135-1-bmwill@google.com","threadId":"47524","inReplyTo":"20180103001828.205012-1-bmwill@google.com","subject":"[PATCH v2 00/27] protocol version 2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:11Z","receivedAt":"2018-01-25T23:58:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v2:\n * Added documentation for fetch\n * changes #defines for state variables to be enums\n * couple code changes to pkt-line functions and documentation\n * Added unit tests for the git-serve binary as well as for ls-refs\n\nAreas for improvement\n * Push isn't implemented, right now this is ok because if v2 is requested the\n server can just default to v0. Before this can be merged we may want to\n change how the client request a new protocol, and not allow for sending\n \"version=2\" when pushing even though the user has it configured. Or maybe\n its fine to just have an older client who doesn't understand how to push\n (and request v2) to die if the server tries to speak v2 at it.\n\n Fixing this essentially would just require piping through a bit more\n information to the function which ultimately runs connect (for both builtins\n and remote-curl)\n\n * I want to make sure that the docs are well written before this gets merged\n so I'm hoping that someone can do a through review on the docs themselves to\n make sure they are clear.\n\n * Right now there is a capability 'stateless-rpc' which essentially makes sure\n that a server command completes after a single round (this is to make sure\n http works cleanly). After talking with some folks it may make more sense\n to just have v2 be stateless in nature so that all commands terminate after\n a single round trip. This makes things a bit easier if a server wants to\n have ssh just be a proxy for http.\n\n One potential thing would be to flip this so that by default the protocol is\n stateless and if a server/command has a state-full mode that can be\n implemented as a capability at a later point. Thoughts?\n\n * Shallow repositories and shallow clones aren't supported yet. I'm working\n on it and it can be either added to v2 by default if people think it needs\n to be in there from the start, or we can add it as a capability at a later\n point.\n\nSeries can also be found on on github: https://github.com/bmwill/git/tree/protocol-v2\n\nBrandon Williams (27):\n pkt-line: introduce packet_read_with_status\n pkt-line: introduce struct packet_reader\n pkt-line: add delim packet support\n upload-pack: convert to a builtin\n upload-pack: factor out processing lines\n transport: use get_refs_via_connect to get refs\n connect: convert get_remote_heads to use struct packet_reader\n connect: discover protocol version outside of get_remote_heads\n transport: store protocol version\n protocol: introduce enum protocol_version value protocol_v2\n test-pkt-line: introduce a packet-line test helper\n serve: introduce git-serve\n ls-refs: introduce ls-refs server command\n connect: request remote refs using v2\n transport: convert get_refs_list to take a list of ref patterns\n transport: convert transport_get_remote_refs to take a list of ref\n patterns\n ls-remote: pass ref patterns when requesting a remote's refs\n fetch: pass ref patterns when fetching\n push: pass ref patterns when pushing\n upload-pack: introduce fetch server command\n fetch-pack: perform a fetch using v2\n transport-helper: remove name parameter\n transport-helper: refactor process_connect_service\n transport-helper: introduce stateless-connect\n pkt-line: add packet_buf_write_len function\n remote-curl: create copy of the service name\n remote-curl: implement stateless-connect command\n\n .gitignore | 1 +\n Documentation/technical/protocol-v2.txt | 270 +++++++++++++++++\n Makefile | 7 +-\n builtin.h | 2 +\n builtin/clone.c | 2 +-\n builtin/fetch-pack.c | 21 +-\n builtin/fetch.c | 14 +-\n builtin/ls-remote.c | 7 +-\n builtin/receive-pack.c | 6 +\n builtin/remote.c | 2 +-\n builtin/send-pack.c | 20 +-\n builtin/serve.c | 30 ++\n builtin/upload-pack.c | 74 +++++\n connect.c | 295 ++++++++++++++-----\n connect.h | 3 +\n fetch-pack.c | 277 +++++++++++++++++-\n fetch-pack.h | 4 +-\n git.c | 2 +\n ls-refs.c | 96 ++++++\n ls-refs.h | 9 +\n pkt-line.c | 149 +++++++++-\n pkt-line.h | 77 +++++\n protocol.c | 2 +\n protocol.h | 1 +\n remote-curl.c | 209 ++++++++++++-\n remote.h | 9 +-\n serve.c | 253 ++++++++++++++++\n serve.h | 15 +\n t/helper/test-pkt-line.c | 62 ++++\n t/t5701-git-serve.sh | 172 +++++++++++\n t/t5702-protocol-v2.sh | 117 ++++++++\n transport-helper.c | 84 +++---\n transport-internal.h | 4 +-\n transport.c | 119 ++++++--\n transport.h | 9 +-\n upload-pack.c | 501 ++++++++++++++++++++++++--------\n upload-pack.h | 18 ++\n 37 files changed, 2646 insertions(+), 297 deletions(-)\n create mode 100644 Documentation/technical/protocol-v2.txt\n create mode 100644 builtin/serve.c\n create mode 100644 builtin/upload-pack.c\n create mode 100644 ls-refs.c\n create mode 100644 ls-refs.h\n create mode 100644 serve.c\n create mode 100644 serve.h\n create mode 100644 t/helper/test-pkt-line.c\n create mode 100755 t/t5701-git-serve.sh\n create mode 100755 t/t5702-protocol-v2.sh\n create mode 100644 upload-pack.h\n\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337422","messageId":"20180125235838.138135-2-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 01/27] pkt-line: introduce packet_read_with_status","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:12Z","receivedAt":"2018-01-25T23:58:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The current pkt-line API encodes the status of a pkt-line read in the\nlength of the read content. An error is indicated with '-1', a flush\nwith '0' (which can be confusing since a return value of '0' can also\nindicate an empty pkt-line), and a positive integer for the length of\nthe read content otherwise. This doesn't leave much room for allowing\nthe addition of additional special packets in the future.\n\nTo solve this introduce 'packet_read_with_status()' which reads a packet\nand returns the status of the read encoded as an 'enum packet_status'\ntype. This allows for easily identifying between special and normal\npackets as well as errors. It also enables easily adding a new special\npacket in the future.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 57 +++++++++++++++++++++++++++++++++++++++++++--------------\n pkt-line.h | 15 +++++++++++++++\n 2 files changed, 58 insertions(+), 14 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 2827ca772..af0d2430f 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -280,28 +280,33 @@ static int packet_length(const char *linelen)\n \treturn (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);\n }\n \n-int packet_read(int fd, char **src_buf, size_t *src_len,\n-\t\tchar *buffer, unsigned size, int options)\n+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n+\t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n+\t\t\t\t\t\tint options)\n {\n-\tint len, ret;\n+\tint len;\n \tchar linelen[4];\n \n-\tret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);\n-\tif (ret < 0)\n-\t\treturn ret;\n+\tif (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)\n+\t\treturn PACKET_READ_EOF;\n+\n \tlen = packet_length(linelen);\n-\tif (len < 0)\n+\n+\tif (len < 0) {\n \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n-\tif (!len) {\n+\t} else if (!len) {\n \t\tpacket_trace(\"0000\", 4, 0);\n-\t\treturn 0;\n+\t\treturn PACKET_READ_FLUSH;\n+\t} else if (len < 4) {\n+\t\tdie(\"protocol error: bad line length %d\", len);\n \t}\n+\n \tlen -= 4;\n-\tif (len >= size)\n+\tif ((unsigned)len >= size)\n \t\tdie(\"protocol error: bad line length %d\", len);\n-\tret = get_packet_data(fd, src_buf, src_len, buffer, len, options);\n-\tif (ret < 0)\n-\t\treturn ret;\n+\n+\tif (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)\n+\t\treturn PACKET_READ_EOF;\n \n \tif ((options & PACKET_READ_CHOMP_NEWLINE) &&\n \t len && buffer[len-1] == '\\n')\n@@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,\n \n \tbuffer[len] = 0;\n \tpacket_trace(buffer, len, 0);\n-\treturn len;\n+\t*pktlen = len;\n+\treturn PACKET_READ_NORMAL;\n+}\n+\n+int packet_read(int fd, char **src_buffer, size_t *src_len,\n+\t\tchar *buffer, unsigned size, int options)\n+{\n+\tenum packet_read_status status;\n+\tint pktlen;\n+\n+\tstatus = packet_read_with_status(fd, src_buffer, src_len,\n+\t\t\t\t\t buffer, size, &pktlen,\n+\t\t\t\t\t options);\n+\tswitch (status) {\n+\tcase PACKET_READ_EOF:\n+\t\tpktlen = -1;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\tbreak;\n+\tcase PACKET_READ_FLUSH:\n+\t\tpktlen = 0;\n+\t\tbreak;\n+\t}\n+\n+\treturn pktlen;\n }\n \n static char *packet_read_line_generic(int fd,\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3dad583e2..06c468927 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n int packet_read(int fd, char **src_buffer, size_t *src_len, char\n \t\t*buffer, unsigned size, int options);\n \n+/*\n+ * Read a packetized line into a buffer like the 'packet_read()' function but\n+ * returns an 'enum packet_read_status' which indicates the status of the read.\n+ * The number of bytes read will be assigined to *pktlen if the status of the\n+ * read was 'PACKET_READ_NORMAL'.\n+ */\n+enum packet_read_status {\n+\tPACKET_READ_EOF = -1,\n+\tPACKET_READ_NORMAL,\n+\tPACKET_READ_FLUSH,\n+};\n+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n+\t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n+\t\t\t\t\t\tint options);\n+\n /*\n * Convenience wrapper for packet_read that is not gentle, and sets the\n * CHOMP_NEWLINE option. The return value is NULL for a flush packet,\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337423","messageId":"20180125235838.138135-3-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 02/27] pkt-line: introduce struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:13Z","receivedAt":"2018-01-25T23:58:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Sometimes it is advantageous to be able to peek the next packet line\nwithout consuming it (e.g. to be able to determine the protocol version\na server is speaking). In order to do that introduce 'struct\npacket_reader' which is an abstraction around the normal packet reading\nlogic. This enables a caller to be able to peek a single line at a time\nusing 'packet_reader_peek()' and having a caller consume a line by\ncalling 'packet_reader_read()'.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n pkt-line.h | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 117 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex af0d2430f..4fc9ad4b0 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -406,3 +406,62 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)\n \t}\n \treturn sb_out->len - orig_len;\n }\n+\n+/* Packet Reader Functions */\n+void packet_reader_init(struct packet_reader *reader, int fd,\n+\t\t\tchar *src_buffer, size_t src_len,\n+\t\t\tint options)\n+{\n+\tmemset(reader, 0, sizeof(*reader));\n+\n+\treader->fd = fd;\n+\treader->src_buffer = src_buffer;\n+\treader->src_len = src_len;\n+\treader->buffer = packet_buffer;\n+\treader->buffer_size = sizeof(packet_buffer);\n+\treader->options = options;\n+}\n+\n+enum packet_read_status packet_reader_read(struct packet_reader *reader)\n+{\n+\tif (reader->line_peeked) {\n+\t\treader->line_peeked = 0;\n+\t\treturn reader->status;\n+\t}\n+\n+\treader->status = packet_read_with_status(reader->fd,\n+\t\t\t\t\t\t &reader->src_buffer,\n+\t\t\t\t\t\t &reader->src_len,\n+\t\t\t\t\t\t reader->buffer,\n+\t\t\t\t\t\t reader->buffer_size,\n+\t\t\t\t\t\t &reader->pktlen,\n+\t\t\t\t\t\t reader->options);\n+\n+\tswitch (reader->status) {\n+\tcase PACKET_READ_EOF:\n+\t\treader->pktlen = -1;\n+\t\treader->line = NULL;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\treader->line = reader->buffer;\n+\t\tbreak;\n+\tcase PACKET_READ_FLUSH:\n+\t\treader->pktlen = 0;\n+\t\treader->line = NULL;\n+\t\tbreak;\n+\t}\n+\n+\treturn reader->status;\n+}\n+\n+enum packet_read_status packet_reader_peek(struct packet_reader *reader)\n+{\n+\t/* Only allow peeking a single line */\n+\tif (reader->line_peeked)\n+\t\treturn reader->status;\n+\n+\t/* Peek a line by reading it and setting peeked flag */\n+\tpacket_reader_read(reader);\n+\treader->line_peeked = 1;\n+\treturn reader->status;\n+}\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 06c468927..7d9f0e537 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n */\n ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);\n \n+struct packet_reader {\n+\t/* source file descriptor */\n+\tint fd;\n+\n+\t/* source buffer and its size */\n+\tchar *src_buffer;\n+\tsize_t src_len;\n+\n+\t/* buffer that pkt-lines are read into and its size */\n+\tchar *buffer;\n+\tunsigned buffer_size;\n+\n+\t/* options to be used during reads */\n+\tint options;\n+\n+\t/* status of the last read */\n+\tenum packet_read_status status;\n+\n+\t/* length of data read during the last read */\n+\tint pktlen;\n+\n+\t/* the last line read */\n+\tconst char *line;\n+\n+\t/* indicates if a line has been peeked */\n+\tint line_peeked;\n+};\n+\n+/*\n+ * Initialize a 'struct packet_reader' object which is an\n+ * abstraction around the 'packet_read_with_status()' function.\n+ */\n+extern void packet_reader_init(struct packet_reader *reader, int fd,\n+\t\t\t char *src_buffer, size_t src_len,\n+\t\t\t int options);\n+\n+/*\n+ * Perform a packet read and return the status of the read.\n+ * The values of 'pktlen' and 'line' are updated based on the status of the\n+ * read as follows:\n+ *\n+ * PACKET_READ_ERROR: 'pktlen' is set to '-1' and 'line' is set to NULL\n+ * PACKET_READ_NORMAL: 'pktlen' is set to the number of bytes read\n+ *\t\t 'line' is set to point at the read line\n+ * PACKET_READ_FLUSH: 'pktlen' is set to '0' and 'line' is set to NULL\n+ */\n+extern enum packet_read_status packet_reader_read(struct packet_reader *reader);\n+\n+/*\n+ * Peek the next packet line without consuming it and return the status.\n+ * The next call to 'packet_reader_read()' will perform a read of the same line\n+ * that was peeked, consuming the line.\n+ *\n+ * Peeking multiple times without calling 'packet_reader_read()' will return\n+ * the same result.\n+ */\n+extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);\n+\n #define DEFAULT_PACKET_MAX 1000\n #define LARGE_PACKET_MAX 65520\n #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337424","messageId":"20180125235838.138135-4-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 03/27] pkt-line: add delim packet support","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:14Z","receivedAt":"2018-01-25T23:58:57Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"One of the design goals of protocol-v2 is to improve the semantics of\nflush packets. Currently in protocol-v1, flush packets are used both to\nindicate a break in a list of packet lines as well as an indication that\none side has finished speaking. This makes it particularly difficult\nto implement proxies as a proxy would need to completely understand git\nprotocol instead of simply looking for a flush packet.\n\nTo do this, introduce the special deliminator packet '0001'. A delim\npacket can then be used as a deliminator between lists of packet lines\nwhile flush packets can be reserved to indicate the end of a response.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 17 +++++++++++++++++\n pkt-line.h | 3 +++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 4fc9ad4b0..726e109ca 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -91,6 +91,12 @@ void packet_flush(int fd)\n \twrite_or_die(fd, \"0000\", 4);\n }\n \n+void packet_delim(int fd)\n+{\n+\tpacket_trace(\"0001\", 4, 1);\n+\twrite_or_die(fd, \"0001\", 4);\n+}\n+\n int packet_flush_gently(int fd)\n {\n \tpacket_trace(\"0000\", 4, 1);\n@@ -105,6 +111,12 @@ void packet_buf_flush(struct strbuf *buf)\n \tstrbuf_add(buf, \"0000\", 4);\n }\n \n+void packet_buf_delim(struct strbuf *buf)\n+{\n+\tpacket_trace(\"0001\", 4, 1);\n+\tstrbuf_add(buf, \"0001\", 4);\n+}\n+\n static void set_packet_header(char *buf, const int size)\n {\n \tstatic char hexchar[] = \"0123456789abcdef\";\n@@ -297,6 +309,9 @@ enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_\n \t} else if (!len) {\n \t\tpacket_trace(\"0000\", 4, 0);\n \t\treturn PACKET_READ_FLUSH;\n+\t} else if (len == 1) {\n+\t\tpacket_trace(\"0001\", 4, 0);\n+\t\treturn PACKET_READ_DELIM;\n \t} else if (len < 4) {\n \t\tdie(\"protocol error: bad line length %d\", len);\n \t}\n@@ -333,6 +348,7 @@ int packet_read(int fd, char **src_buffer, size_t *src_len,\n \t\tbreak;\n \tcase PACKET_READ_NORMAL:\n \t\tbreak;\n+\tcase PACKET_READ_DELIM:\n \tcase PACKET_READ_FLUSH:\n \t\tpktlen = 0;\n \t\tbreak;\n@@ -445,6 +461,7 @@ enum packet_read_status packet_reader_read(struct packet_reader *reader)\n \tcase PACKET_READ_NORMAL:\n \t\treader->line = reader->buffer;\n \t\tbreak;\n+\tcase PACKET_READ_DELIM:\n \tcase PACKET_READ_FLUSH:\n \t\treader->pktlen = 0;\n \t\treader->line = NULL;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 7d9f0e537..16fe8bdbf 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -20,8 +20,10 @@\n * side can't, we stay with pure read/write interfaces.\n */\n void packet_flush(int fd);\n+void packet_delim(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n+void packet_buf_delim(struct strbuf *buf);\n void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n@@ -75,6 +77,7 @@ enum packet_read_status {\n \tPACKET_READ_EOF = -1,\n \tPACKET_READ_NORMAL,\n \tPACKET_READ_FLUSH,\n+\tPACKET_READ_DELIM,\n };\n enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n \t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337425","messageId":"20180125235838.138135-5-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 04/27] upload-pack: convert to a builtin","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:15Z","receivedAt":"2018-01-25T23:59:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to allow for code sharing with the server-side of fetch in\nprotocol-v2 convert upload-pack to be a builtin.\n\nSigned-off-by: Brandon Williams \n---\n Makefile | 3 +-\n builtin.h | 1 +\n builtin/upload-pack.c | 67 +++++++++++++++++++++++++++++++\n git.c | 1 +\n upload-pack.c | 107 ++++++++++++--------------------------------------\n upload-pack.h | 13 ++++++\n 6 files changed, 109 insertions(+), 83 deletions(-)\n create mode 100644 builtin/upload-pack.c\n create mode 100644 upload-pack.h\n\ndiff --git a/Makefile b/Makefile\nindex 1a9b23b67..b7ccc05fa 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -639,7 +639,6 @@ PROGRAM_OBJS += imap-send.o\n PROGRAM_OBJS += sh-i18n--envsubst.o\n PROGRAM_OBJS += shell.o\n PROGRAM_OBJS += show-index.o\n-PROGRAM_OBJS += upload-pack.o\n PROGRAM_OBJS += remote-testsvn.o\n \n # Binary suffix, set to .exe for Windows builds\n@@ -909,6 +908,7 @@ LIB_OBJS += tree-diff.o\n LIB_OBJS += tree.o\n LIB_OBJS += tree-walk.o\n LIB_OBJS += unpack-trees.o\n+LIB_OBJS += upload-pack.o\n LIB_OBJS += url.o\n LIB_OBJS += urlmatch.o\n LIB_OBJS += usage.o\n@@ -1026,6 +1026,7 @@ BUILTIN_OBJS += builtin/update-index.o\n BUILTIN_OBJS += builtin/update-ref.o\n BUILTIN_OBJS += builtin/update-server-info.o\n BUILTIN_OBJS += builtin/upload-archive.o\n+BUILTIN_OBJS += builtin/upload-pack.o\n BUILTIN_OBJS += builtin/var.o\n BUILTIN_OBJS += builtin/verify-commit.o\n BUILTIN_OBJS += builtin/verify-pack.o\ndiff --git a/builtin.h b/builtin.h\nindex 42378f3aa..f332a1257 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -231,6 +231,7 @@ extern int cmd_update_ref(int argc, const char **argv, const char *prefix);\n extern int cmd_update_server_info(int argc, const char **argv, const char *prefix);\n extern int cmd_upload_archive(int argc, const char **argv, const char *prefix);\n extern int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix);\n+extern int cmd_upload_pack(int argc, const char **argv, const char *prefix);\n extern int cmd_var(int argc, const char **argv, const char *prefix);\n extern int cmd_verify_commit(int argc, const char **argv, const char *prefix);\n extern int cmd_verify_tag(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\nnew file mode 100644\nindex 000000000..2cb5cb35b\n--- /dev/null\n+++ b/builtin/upload-pack.c\n@@ -0,0 +1,67 @@\n+#include \"cache.h\"\n+#include \"builtin.h\"\n+#include \"exec_cmd.h\"\n+#include \"pkt-line.h\"\n+#include \"parse-options.h\"\n+#include \"protocol.h\"\n+#include \"upload-pack.h\"\n+\n+static const char * const upload_pack_usage[] = {\n+\tN_(\"git upload-pack [] \"),\n+\tNULL\n+};\n+\n+int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *dir;\n+\tint strict = 0;\n+\tstruct upload_pack_options opts = { 0 };\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n+\t\t\t N_(\"quit after a single request/response exchange\")),\n+\t\tOPT_BOOL(0, \"advertise-refs\", &opts.advertise_refs,\n+\t\t\t N_(\"exit immediately after initial ref advertisement\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict,\n+\t\t\t N_(\"do not try /.git/ if is no Git directory\")),\n+\t\tOPT_INTEGER(0, \"timeout\", &opts.timeout,\n+\t\t\t N_(\"interrupt transfer after seconds of inactivity\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tpacket_trace_identity(\"upload-pack\");\n+\tcheck_replace_refs = 0;\n+\n+\targc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);\n+\n+\tif (argc != 1)\n+\t\tusage_with_options(upload_pack_usage, options);\n+\n+\tif (opts.timeout)\n+\t\topts.daemon_mode = 1;\n+\n+\tsetup_path();\n+\n+\tdir = argv[0];\n+\n+\tif (!enter_repo(dir, strict))\n+\t\tdie(\"'%s' does not appear to be a git repository\", dir);\n+\n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\t\tif (opts.advertise_refs || !opts.stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\n+\t\t/* fallthrough */\n+\tcase protocol_v0:\n+\t\tupload_pack(&opts);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex c870b9719..f71073dc8 100644\n--- a/git.c\n+++ b/git.c\n@@ -478,6 +478,7 @@ static struct cmd_struct commands[] = {\n \t{ \"update-server-info\", cmd_update_server_info, RUN_SETUP },\n \t{ \"upload-archive\", cmd_upload_archive },\n \t{ \"upload-archive--writer\", cmd_upload_archive_writer },\n+\t{ \"upload-pack\", cmd_upload_pack },\n \t{ \"var\", cmd_var, RUN_SETUP_GENTLY },\n \t{ \"verify-commit\", cmd_verify_commit, RUN_SETUP },\n \t{ \"verify-pack\", cmd_verify_pack },\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d5de18127..2ad73a98b 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -6,7 +6,6 @@\n #include \"tag.h\"\n #include \"object.h\"\n #include \"commit.h\"\n-#include \"exec_cmd.h\"\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n@@ -15,15 +14,10 @@\n #include \"sigchain.h\"\n #include \"version.h\"\n #include \"string-list.h\"\n-#include \"parse-options.h\"\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n #include \"protocol.h\"\n-\n-static const char * const upload_pack_usage[] = {\n-\tN_(\"git upload-pack [] \"),\n-\tNULL\n-};\n+#include \"upload-pack.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -61,7 +55,6 @@ static int keepalive = 5;\n * otherwise maximum packet size (up to 65520 bytes).\n */\n static int use_sideband;\n-static int advertise_refs;\n static int stateless_rpc;\n static const char *pack_objects_hook;\n \n@@ -977,33 +970,6 @@ static int find_symref(const char *refname, const struct object_id *oid,\n \treturn 0;\n }\n \n-static void upload_pack(void)\n-{\n-\tstruct string_list symref = STRING_LIST_INIT_DUP;\n-\n-\thead_ref_namespaced(find_symref, &symref);\n-\n-\tif (advertise_refs || !stateless_rpc) {\n-\t\treset_timeout();\n-\t\thead_ref_namespaced(send_ref, &symref);\n-\t\tfor_each_namespaced_ref(send_ref, &symref);\n-\t\tadvertise_shallow_grafts(1);\n-\t\tpacket_flush(1);\n-\t} else {\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n-\t}\n-\tstring_list_clear(&symref, 1);\n-\tif (advertise_refs)\n-\t\treturn;\n-\n-\treceive_needs();\n-\tif (want_obj.nr) {\n-\t\tget_common_commits();\n-\t\tcreate_pack_file();\n-\t}\n-}\n-\n static int upload_pack_config(const char *var, const char *value, void *unused)\n {\n \tif (!strcmp(\"uploadpack.allowtipsha1inwant\", var)) {\n@@ -1032,58 +998,35 @@ static int upload_pack_config(const char *var, const char *value, void *unused)\n \treturn parse_hide_refs_config(var, value, \"uploadpack\");\n }\n \n-int cmd_main(int argc, const char **argv)\n+void upload_pack(struct upload_pack_options *options)\n {\n-\tconst char *dir;\n-\tint strict = 0;\n-\tstruct option options[] = {\n-\t\tOPT_BOOL(0, \"stateless-rpc\", &stateless_rpc,\n-\t\t\t N_(\"quit after a single request/response exchange\")),\n-\t\tOPT_BOOL(0, \"advertise-refs\", &advertise_refs,\n-\t\t\t N_(\"exit immediately after initial ref advertisement\")),\n-\t\tOPT_BOOL(0, \"strict\", &strict,\n-\t\t\t N_(\"do not try /.git/ if is no Git directory\")),\n-\t\tOPT_INTEGER(0, \"timeout\", &timeout,\n-\t\t\t N_(\"interrupt transfer after seconds of inactivity\")),\n-\t\tOPT_END()\n-\t};\n-\n-\tpacket_trace_identity(\"upload-pack\");\n-\tcheck_replace_refs = 0;\n-\n-\targc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);\n-\n-\tif (argc != 1)\n-\t\tusage_with_options(upload_pack_usage, options);\n-\n-\tif (timeout)\n-\t\tdaemon_mode = 1;\n-\n-\tsetup_path();\n-\n-\tdir = argv[0];\n+\tstruct string_list symref = STRING_LIST_INIT_DUP;\n \n-\tif (!enter_repo(dir, strict))\n-\t\tdie(\"'%s' does not appear to be a git repository\", dir);\n+\tstateless_rpc = options->stateless_rpc;\n+\ttimeout = options->timeout;\n+\tdaemon_mode = options->daemon_mode;\n \n \tgit_config(upload_pack_config, NULL);\n \n-\tswitch (determine_protocol_version_server()) {\n-\tcase protocol_v1:\n-\t\t/*\n-\t\t * v1 is just the original protocol with a version string,\n-\t\t * so just fall through after writing the version string.\n-\t\t */\n-\t\tif (advertise_refs || !stateless_rpc)\n-\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n-\n-\t\t/* fallthrough */\n-\tcase protocol_v0:\n-\t\tupload_pack();\n-\t\tbreak;\n-\tcase protocol_unknown_version:\n-\t\tBUG(\"unknown protocol version\");\n+\thead_ref_namespaced(find_symref, &symref);\n+\n+\tif (options->advertise_refs || !stateless_rpc) {\n+\t\treset_timeout();\n+\t\thead_ref_namespaced(send_ref, &symref);\n+\t\tfor_each_namespaced_ref(send_ref, &symref);\n+\t\tadvertise_shallow_grafts(1);\n+\t\tpacket_flush(1);\n+\t} else {\n+\t\thead_ref_namespaced(check_ref, NULL);\n+\t\tfor_each_namespaced_ref(check_ref, NULL);\n \t}\n+\tstring_list_clear(&symref, 1);\n+\tif (options->advertise_refs)\n+\t\treturn;\n \n-\treturn 0;\n+\treceive_needs();\n+\tif (want_obj.nr) {\n+\t\tget_common_commits();\n+\t\tcreate_pack_file();\n+\t}\n }\ndiff --git a/upload-pack.h b/upload-pack.h\nnew file mode 100644\nindex 000000000..a71e4dc7e\n--- /dev/null\n+++ b/upload-pack.h\n@@ -0,0 +1,13 @@\n+#ifndef UPLOAD_PACK_H\n+#define UPLOAD_PACK_H\n+\n+struct upload_pack_options {\n+\tint stateless_rpc;\n+\tint advertise_refs;\n+\tunsigned int timeout;\n+\tint daemon_mode;\n+};\n+\n+void upload_pack(struct upload_pack_options *options);\n+\n+#endif /* UPLOAD_PACK_H */\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337426","messageId":"20180125235838.138135-7-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 06/27] transport: use get_refs_via_connect to get refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:17Z","receivedAt":"2018-01-25T23:59:04Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Remove code duplication and use the existing 'get_refs_via_connect()'\nfunction to retrieve a remote's heads in 'fetch_refs_via_pack()' and\n'git_transport_push()'.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 18 ++++--------------\n 1 file changed, 4 insertions(+), 14 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex fc802260f..8e8779096 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n \n-\tif (!data->got_remote_heads) {\n-\t\tconnect_setup(transport, 0);\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL, &data->shallow);\n-\t\tdata->got_remote_heads = 1;\n-\t}\n+\tif (!data->got_remote_heads)\n+\t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n \trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n@@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tstruct send_pack_args args;\n \tint ret;\n \n-\tif (!data->got_remote_heads) {\n-\t\tstruct ref *tmp_refs;\n-\t\tconnect_setup(transport, 1);\n-\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n-\t\t\t\t NULL, &data->shallow);\n-\t\tdata->got_remote_heads = 1;\n-\t}\n+\tif (!data->got_remote_heads)\n+\t\tget_refs_via_connect(transport, 1);\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337427","messageId":"20180125235838.138135-8-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 07/27] connect: convert get_remote_heads to use struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:18Z","receivedAt":"2018-01-25T23:59:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to allow for better control flow when protocol_v2 is introduced\nconvert 'get_remote_heads()' to use 'struct packet_reader' to read\npacket lines. This enables a client to be able to peek the first line\nof a server's response (without consuming it) in order to determine the\nprotocol version its speaking and then passing control to the\nappropriate handler.\n\nThis is needed because the initial response from a server speaking\nprotocol_v0 includes the first ref, while subsequent protocol versions\nrespond with a version line. We want to be able to read this first line\nwithout consuming the first ref sent in the protocol_v0 case so that the\nprotocol version the server is speaking can be determined outside of\n'get_remote_heads()' in a future patch.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 174 ++++++++++++++++++++++++++++++++++----------------------------\n 1 file changed, 96 insertions(+), 78 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex c3a014c5b..00e90075c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -48,6 +48,12 @@ int check_ref_type(const struct ref *ref, int flags)\n \n static void die_initial_contact(int unexpected)\n {\n+\t/*\n+\t * A hang-up after seeing some response from the other end\n+\t * means that it is unexpected, as we know the other end is\n+\t * willing to talk to us. A hang-up before seeing any\n+\t * response does not necessarily mean an ACL problem, though.\n+\t */\n \tif (unexpected)\n \t\tdie(_(\"The remote end hung up upon initial contact\"));\n \telse\n@@ -56,6 +62,41 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n+static enum protocol_version discover_version(struct packet_reader *reader)\n+{\n+\tenum protocol_version version = protocol_unknown_version;\n+\n+\t/*\n+\t * Peek the first line of the server's response to\n+\t * determine the protocol version the server is speaking.\n+\t */\n+\tswitch (packet_reader_peek(reader)) {\n+\tcase PACKET_READ_EOF:\n+\t\tdie_initial_contact(0);\n+\tcase PACKET_READ_FLUSH:\n+\tcase PACKET_READ_DELIM:\n+\t\tversion = protocol_v0;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\tversion = determine_protocol_version_client(reader->line);\n+\t\tbreak;\n+\t}\n+\n+\t/* Maybe process capabilities here, at least for v2 */\n+\tswitch (version) {\n+\tcase protocol_v1:\n+\t\t/* Read the peeked version line */\n+\t\tpacket_reader_read(reader);\n+\t\tbreak;\n+\tcase protocol_v0:\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tdie(\"unknown protocol version: '%s'\\n\", reader->line);\n+\t}\n+\n+\treturn version;\n+}\n+\n static void parse_one_symref_info(struct string_list *symref, const char *val, int len)\n {\n \tchar *sym, *target;\n@@ -109,60 +150,21 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n-/*\n- * Read one line of a server's ref advertisement into packet_buffer.\n- */\n-static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n-\t\t\t int *responded)\n+static void process_capabilities(const char *line, int *len)\n {\n-\tint len = packet_read(in, src_buf, src_len,\n-\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\tconst char *arg;\n-\tif (len < 0)\n-\t\tdie_initial_contact(*responded);\n-\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n-\t\tdie(\"remote error: %s\", arg);\n-\n-\t*responded = 1;\n-\n-\treturn len;\n-}\n-\n-#define EXPECTING_PROTOCOL_VERSION 0\n-#define EXPECTING_FIRST_REF 1\n-#define EXPECTING_REF 2\n-#define EXPECTING_SHALLOW 3\n-\n-/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n-static int process_protocol_version(void)\n-{\n-\tswitch (determine_protocol_version_client(packet_buffer)) {\n-\tcase protocol_v1:\n-\t\treturn 1;\n-\tcase protocol_v0:\n-\t\treturn 0;\n-\tdefault:\n-\t\tdie(\"server is speaking an unknown protocol\");\n-\t}\n-}\n-\n-static void process_capabilities(int *len)\n-{\n-\tint nul_location = strlen(packet_buffer);\n+\tint nul_location = strlen(line);\n \tif (nul_location == *len)\n \t\treturn;\n-\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n+\tserver_capabilities = xstrdup(line + nul_location + 1);\n \t*len = nul_location;\n }\n \n-static int process_dummy_ref(void)\n+static int process_dummy_ref(const char *line)\n {\n \tstruct object_id oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(packet_buffer, &oid, &name))\n+\tif (parse_oid_hex(line, &oid, &name))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -171,20 +173,20 @@ static int process_dummy_ref(void)\n \treturn !oidcmp(&null_oid, &oid) && !strcmp(name, \"capabilities^{}\");\n }\n \n-static void check_no_capabilities(int len)\n+static void check_no_capabilities(const char *line, int len)\n {\n-\tif (strlen(packet_buffer) != len)\n+\tif (strlen(line) != len)\n \t\twarning(\"Ignoring capabilities after first line '%s'\",\n-\t\t\tpacket_buffer + strlen(packet_buffer));\n+\t\t\tline + strlen(line));\n }\n \n-static int process_ref(int len, struct ref ***list, unsigned int flags,\n-\t\t struct oid_array *extra_have)\n+static int process_ref(const char *line, int len, struct ref ***list,\n+\t\t unsigned int flags, struct oid_array *extra_have)\n {\n \tstruct object_id old_oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n+\tif (parse_oid_hex(line, &old_oid, &name))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -200,16 +202,17 @@ static int process_ref(int len, struct ref ***list, unsigned int flags,\n \t\t**list = ref;\n \t\t*list = &ref->next;\n \t}\n-\tcheck_no_capabilities(len);\n+\tcheck_no_capabilities(line, len);\n \treturn 1;\n }\n \n-static int process_shallow(int len, struct oid_array *shallow_points)\n+static int process_shallow(const char *line, int len,\n+\t\t\t struct oid_array *shallow_points)\n {\n \tconst char *arg;\n \tstruct object_id old_oid;\n \n-\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\tif (!skip_prefix(line, \"shallow \", &arg))\n \t\treturn 0;\n \n \tif (get_oid_hex(arg, &old_oid))\n@@ -217,10 +220,17 @@ static int process_shallow(int len, struct oid_array *shallow_points)\n \tif (!shallow_points)\n \t\tdie(\"repository on the other end cannot be shallow\");\n \toid_array_append(shallow_points, &old_oid);\n-\tcheck_no_capabilities(len);\n+\tcheck_no_capabilities(line, len);\n \treturn 1;\n }\n \n+enum get_remote_heads_state {\n+\tEXPECTING_FIRST_REF = 0,\n+\tEXPECTING_REF,\n+\tEXPECTING_SHALLOW,\n+\tEXPECTING_DONE,\n+};\n+\n /*\n * Read all the refs from the other end\n */\n@@ -230,47 +240,55 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t struct oid_array *shallow_points)\n {\n \tstruct ref **orig_list = list;\n+\tint len = 0;\n+\tenum get_remote_heads_state state = EXPECTING_FIRST_REF;\n+\tstruct packet_reader reader;\n+\tconst char *arg;\n \n-\t/*\n-\t * A hang-up after seeing some response from the other end\n-\t * means that it is unexpected, as we know the other end is\n-\t * willing to talk to us. A hang-up before seeing any\n-\t * response does not necessarily mean an ACL problem, though.\n-\t */\n-\tint responded = 0;\n-\tint len;\n-\tint state = EXPECTING_PROTOCOL_VERSION;\n+\tpacket_reader_init(&reader, in, src_buf, src_len,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tdiscover_version(&reader);\n \n \t*list = NULL;\n \n-\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\twhile (state != EXPECTING_DONE) {\n+\t\tswitch (packet_reader_read(&reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tdie_initial_contact(1);\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tlen = reader.pktlen;\n+\t\t\tif (len > 4 && skip_prefix(reader.line, \"ERR \", &arg))\n+\t\t\t\tdie(\"remote error: %s\", arg);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tstate = EXPECTING_DONE;\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tdie(\"invalid packet\");\n+\t\t}\n+\n \t\tswitch (state) {\n-\t\tcase EXPECTING_PROTOCOL_VERSION:\n-\t\t\tif (process_protocol_version()) {\n-\t\t\t\tstate = EXPECTING_FIRST_REF;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tstate = EXPECTING_FIRST_REF;\n-\t\t\t/* fallthrough */\n \t\tcase EXPECTING_FIRST_REF:\n-\t\t\tprocess_capabilities(&len);\n-\t\t\tif (process_dummy_ref()) {\n+\t\t\tprocess_capabilities(reader.line, &len);\n+\t\t\tif (process_dummy_ref(reader.line)) {\n \t\t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tstate = EXPECTING_REF;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_REF:\n-\t\t\tif (process_ref(len, &list, flags, extra_have))\n+\t\t\tif (process_ref(reader.line, len, &list, flags, extra_have))\n \t\t\t\tbreak;\n \t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_SHALLOW:\n-\t\t\tif (process_shallow(len, shallow_points))\n+\t\t\tif (process_shallow(reader.line, len, shallow_points))\n \t\t\t\tbreak;\n-\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n-\t\tdefault:\n-\t\t\tdie(\"unexpected state %d\", state);\n+\t\t\tdie(\"protocol error: unexpected '%s'\", reader.line);\n+\t\tcase EXPECTING_DONE:\n+\t\t\tbreak;\n \t\t}\n \t}\n \n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337428","messageId":"20180125235838.138135-9-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 08/27] connect: discover protocol version outside of get_remote_heads","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:19Z","receivedAt":"2018-01-25T23:59:08Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to prepare for the addition of protocol_v2 push the protocol\nversion discovery outside of 'get_remote_heads()'. This will allow for\nkeeping the logic for processing the reference advertisement for\nprotocol_v1 and protocol_v0 separate from the logic for protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 16 +++++++++++++++-\n builtin/send-pack.c | 17 +++++++++++++++--\n connect.c | 27 ++++++++++-----------------\n connect.h | 3 +++\n remote-curl.c | 20 ++++++++++++++++++--\n remote.h | 5 +++--\n transport.c | 24 +++++++++++++++++++-----\n 7 files changed, 83 insertions(+), 29 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 366b9d13f..85d4faf76 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -4,6 +4,7 @@\n #include \"remote.h\"\n #include \"connect.h\"\n #include \"sha1-array.h\"\n+#include \"protocol.h\"\n \n static const char fetch_pack_usage[] =\n \"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] \"\n@@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tstruct fetch_pack_args args;\n \tstruct oid_array shallow = OID_ARRAY_INIT;\n \tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n+\tstruct packet_reader reader;\n \n \tpacket_trace_identity(\"fetch-pack\");\n \n@@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\tif (!conn)\n \t\t\treturn args.diag_url ? 0 : 1;\n \t}\n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n+\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n \t\t\t &shallow, pack_lockfile_ptr);\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex fc4f0bb5f..83cb125a6 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -14,6 +14,7 @@\n #include \"sha1-array.h\"\n #include \"gpg-interface.h\"\n #include \"gettext.h\"\n+#include \"protocol.h\"\n \n static const char * const send_pack_usage[] = {\n \tN_(\"git send-pack [--all | --mirror] [--dry-run] [--force] \"\n@@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tstruct packet_reader reader;\n \n \tstruct option options[] = {\n \t\tOPT__VERBOSITY(&verbose),\n@@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n-\t\t\t &extra_have, &shallow);\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n+\t\t\t\t &extra_have, &shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/connect.c b/connect.c\nindex 00e90075c..db3c9d24c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n-static enum protocol_version discover_version(struct packet_reader *reader)\n+enum protocol_version discover_version(struct packet_reader *reader)\n {\n \tenum protocol_version version = protocol_unknown_version;\n \n@@ -234,7 +234,7 @@ enum get_remote_heads_state {\n /*\n * Read all the refs from the other end\n */\n-struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n+struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t struct ref **list, unsigned int flags,\n \t\t\t struct oid_array *extra_have,\n \t\t\t struct oid_array *shallow_points)\n@@ -242,24 +242,17 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \tstruct ref **orig_list = list;\n \tint len = 0;\n \tenum get_remote_heads_state state = EXPECTING_FIRST_REF;\n-\tstruct packet_reader reader;\n \tconst char *arg;\n \n-\tpacket_reader_init(&reader, in, src_buf, src_len,\n-\t\t\t PACKET_READ_CHOMP_NEWLINE |\n-\t\t\t PACKET_READ_GENTLE_ON_EOF);\n-\n-\tdiscover_version(&reader);\n-\n \t*list = NULL;\n \n \twhile (state != EXPECTING_DONE) {\n-\t\tswitch (packet_reader_read(&reader)) {\n+\t\tswitch (packet_reader_read(reader)) {\n \t\tcase PACKET_READ_EOF:\n \t\t\tdie_initial_contact(1);\n \t\tcase PACKET_READ_NORMAL:\n-\t\t\tlen = reader.pktlen;\n-\t\t\tif (len > 4 && skip_prefix(reader.line, \"ERR \", &arg))\n+\t\t\tlen = reader->pktlen;\n+\t\t\tif (len > 4 && skip_prefix(reader->line, \"ERR \", &arg))\n \t\t\t\tdie(\"remote error: %s\", arg);\n \t\t\tbreak;\n \t\tcase PACKET_READ_FLUSH:\n@@ -271,22 +264,22 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \n \t\tswitch (state) {\n \t\tcase EXPECTING_FIRST_REF:\n-\t\t\tprocess_capabilities(reader.line, &len);\n-\t\t\tif (process_dummy_ref(reader.line)) {\n+\t\t\tprocess_capabilities(reader->line, &len);\n+\t\t\tif (process_dummy_ref(reader->line)) {\n \t\t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tstate = EXPECTING_REF;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_REF:\n-\t\t\tif (process_ref(reader.line, len, &list, flags, extra_have))\n+\t\t\tif (process_ref(reader->line, len, &list, flags, extra_have))\n \t\t\t\tbreak;\n \t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_SHALLOW:\n-\t\t\tif (process_shallow(reader.line, len, shallow_points))\n+\t\t\tif (process_shallow(reader->line, len, shallow_points))\n \t\t\t\tbreak;\n-\t\t\tdie(\"protocol error: unexpected '%s'\", reader.line);\n+\t\t\tdie(\"protocol error: unexpected '%s'\", reader->line);\n \t\tcase EXPECTING_DONE:\n \t\t\tbreak;\n \t\t}\ndiff --git a/connect.h b/connect.h\nindex 01f14cdf3..cdb8979dc 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);\n extern const char *server_feature_value(const char *feature, int *len_ret);\n extern int url_is_local_not_ssh(const char *url);\n \n+struct packet_reader;\n+extern enum protocol_version discover_version(struct packet_reader *reader);\n+\n #endif\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 0053b0954..9f6d07683 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"remote.h\"\n+#include \"connect.h\"\n #include \"strbuf.h\"\n #include \"walker.h\"\n #include \"http.h\"\n@@ -13,6 +14,7 @@\n #include \"credential.h\"\n #include \"sha1-array.h\"\n #include \"send-pack.h\"\n+#include \"protocol.h\"\n \n static struct remote *remote;\n /* always ends with a trailing slash */\n@@ -176,8 +178,22 @@ static struct discovery *last_discovery;\n static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n-\tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n+\tstruct packet_reader reader;\n+\n+\tpacket_reader_init(&reader, -1, heads->buf, heads->len,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n+\t\t\t\t NULL, &heads->shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \treturn list;\n }\n \ndiff --git a/remote.h b/remote.h\nindex 1f6611be2..2016461df 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n void free_refs(struct ref *ref);\n \n struct oid_array;\n-extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n+struct packet_reader;\n+extern struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t\t struct ref **list, unsigned int flags,\n \t\t\t\t struct oid_array *extra_have,\n-\t\t\t\t struct oid_array *shallow);\n+\t\t\t\t struct oid_array *shallow_points);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\ndiff --git a/transport.c b/transport.c\nindex 8e8779096..63c3dbab9 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -18,6 +18,7 @@\n #include \"sha1-array.h\"\n #include \"sigchain.h\"\n #include \"transport-internal.h\"\n+#include \"protocol.h\"\n \n static void set_upstreams(struct transport *transport, struct ref *refs,\n \tint pretend)\n@@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)\n static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n {\n \tstruct git_transport_data *data = transport->data;\n-\tstruct ref *refs;\n+\tstruct ref *refs = NULL;\n+\tstruct packet_reader reader;\n \n \tconnect_setup(transport, for_push);\n-\tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0,\n-\t\t\t &data->extra_have,\n-\t\t\t &data->shallow);\n+\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &refs,\n+\t\t\t\t for_push ? REF_NORMAL : 0,\n+\t\t\t\t &data->extra_have,\n+\t\t\t\t &data->shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337429","messageId":"20180125235838.138135-10-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 09/27] transport: store protocol version","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:20Z","receivedAt":"2018-01-25T23:59:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Once protocol_v2 is introduced requesting a fetch or a push will need to\nbe handled differently depending on the protocol version. Store the\nprotocol version the server is speaking in 'struct git_transport_data'\nand use it to determine what to do in the case of a fetch or a push.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 35 ++++++++++++++++++++++++++---------\n 1 file changed, 26 insertions(+), 9 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex 63c3dbab9..2378dcb38 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -118,6 +118,7 @@ struct git_transport_data {\n \tstruct child_process *conn;\n \tint fd[2];\n \tunsigned got_remote_heads : 1;\n+\tenum protocol_version version;\n \tstruct oid_array extra_have;\n \tstruct oid_array shallow;\n };\n@@ -200,7 +201,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n-\tswitch (discover_version(&reader)) {\n+\tdata->version = discover_version(&reader);\n+\tswitch (data->version) {\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &refs,\n@@ -221,7 +223,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n {\n \tint ret = 0;\n \tstruct git_transport_data *data = transport->data;\n-\tstruct ref *refs;\n+\tstruct ref *refs = NULL;\n \tchar *dest = xstrdup(transport->url);\n \tstruct fetch_pack_args args;\n \tstruct ref *refs_tmp = NULL;\n@@ -247,10 +249,18 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tif (!data->got_remote_heads)\n \t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n-\trefs = fetch_pack(&args, data->fd, data->conn,\n-\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n-\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n-\t\t\t &transport->pack_lockfile);\n+\tswitch (data->version) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\trefs = fetch_pack(&args, data->fd, data->conn,\n+\t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n+\t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n+\t\t\t\t &transport->pack_lockfile);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \tclose(data->fd[0]);\n \tclose(data->fd[1]);\n \tif (finish_connect(data->conn))\n@@ -549,7 +559,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n {\n \tstruct git_transport_data *data = transport->data;\n \tstruct send_pack_args args;\n-\tint ret;\n+\tint ret = 0;\n \n \tif (!data->got_remote_heads)\n \t\tget_refs_via_connect(transport, 1);\n@@ -574,8 +584,15 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \telse\n \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n \n-\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n-\t\t\t&data->extra_have);\n+\tswitch (data->version) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n+\t\t\t\t&data->extra_have);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \tclose(data->fd[1]);\n \tclose(data->fd[0]);\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337430","messageId":"20180125235838.138135-11-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:21Z","receivedAt":"2018-01-25T23:59:17Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce protocol_v2, a new value for 'enum protocol_version'.\nSubsequent patches will fill in the implementation of protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 3 +++\n builtin/receive-pack.c | 6 ++++++\n builtin/send-pack.c | 3 +++\n builtin/upload-pack.c | 7 +++++++\n connect.c | 3 +++\n protocol.c | 2 ++\n protocol.h | 1 +\n remote-curl.c | 3 +++\n transport.c | 9 +++++++++\n 9 files changed, 37 insertions(+)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 85d4faf76..f492e8abd 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex b7ce7c7f5..3656e94fd 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tunpack_limit = receive_unpack_limit;\n \n \tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v2:\n+\t\t/*\n+\t\t * push support for protocol v2 has not been implemented yet,\n+\t\t * so ignore the request to use v2 and fallback to using v0.\n+\t\t */\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 83cb125a6..b5427f75e 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\ndiff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\nindex 2cb5cb35b..8d53e9794 100644\n--- a/builtin/upload-pack.c\n+++ b/builtin/upload-pack.c\n@@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n \tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v2:\n+\t\t/*\n+\t\t * fetch support for protocol v2 has not been implemented yet,\n+\t\t * so ignore the request to use v2 and fallback to using v0.\n+\t\t */\n+\t\tupload_pack(&opts);\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\ndiff --git a/connect.c b/connect.c\nindex db3c9d24c..f2157a821 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)\n \n \t/* Maybe process capabilities here, at least for v2 */\n \tswitch (version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/* Read the peeked version line */\n \t\tpacket_reader_read(reader);\ndiff --git a/protocol.c b/protocol.c\nindex 43012b7eb..5e636785d 100644\n--- a/protocol.c\n+++ b/protocol.c\n@@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)\n \t\treturn protocol_v0;\n \telse if (!strcmp(value, \"1\"))\n \t\treturn protocol_v1;\n+\telse if (!strcmp(value, \"2\"))\n+\t\treturn protocol_v2;\n \telse\n \t\treturn protocol_unknown_version;\n }\ndiff --git a/protocol.h b/protocol.h\nindex 1b2bc94a8..2ad35e433 100644\n--- a/protocol.h\n+++ b/protocol.h\n@@ -5,6 +5,7 @@ enum protocol_version {\n \tprotocol_unknown_version = -1,\n \tprotocol_v0 = 0,\n \tprotocol_v1 = 1,\n+\tprotocol_v2 = 2,\n };\n \n /*\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 9f6d07683..dae8a4a48 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\ndiff --git a/transport.c b/transport.c\nindex 2378dcb38..83d9dd1df 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &refs,\n@@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\trefs = fetch_pack(&args, data->fd, data->conn,\n@@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n \n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337431","messageId":"20180125235838.138135-13-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 12/27] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:23Z","receivedAt":"2018-01-25T23:59:19Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce git-serve, the base server for protocol version 2.\n\nProtocol version 2 is intended to be a replacement for Git's current\nwire protocol. The intention is that it will be a simpler, less\nwasteful protocol which can evolve over time.\n\nProtocol version 2 improves upon version 1 by eliminating the initial\nref advertisement. In its place a server will export a list of\ncapabilities and commands which it supports in a capability\nadvertisement. A client can then request that a particular command be\nexecuted by providing a number of capabilities and command specific\nparameters. At the completion of a command, a client can request that\nanother command be executed or can terminate the connection by sending a\nflush packet.\n\nSigned-off-by: Brandon Williams \n---\n .gitignore | 1 +\n Documentation/technical/protocol-v2.txt | 117 +++++++++++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/serve.c | 30 ++++\n git.c | 1 +\n serve.c | 249 ++++++++++++++++++++++++++++++++\n serve.h | 15 ++\n t/t5701-git-serve.sh | 56 +++++++\n 9 files changed, 472 insertions(+)\n create mode 100644 Documentation/technical/protocol-v2.txt\n create mode 100644 builtin/serve.c\n create mode 100644 serve.c\n create mode 100644 serve.h\n create mode 100755 t/t5701-git-serve.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 833ef3b0b..2d0450c26 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -140,6 +140,7 @@\n /git-rm\n /git-send-email\n /git-send-pack\n+/git-serve\n /git-sh-i18n\n /git-sh-i18n--envsubst\n /git-sh-setup\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nnew file mode 100644\nindex 000000000..7f619a76c\n--- /dev/null\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -0,0 +1,117 @@\n+ Git Wire Protocol, Version 2\n+==============================\n+\n+This document presents a specification for a version 2 of Git's wire\n+protocol. Protocol v2 will improve upon v1 in the following ways:\n+\n+ * Instead of multiple service names, multiple commands will be\n+ supported by a single service.\n+ * Easily extendable as capabilities are moved into their own section\n+ of the protocol, no longer being hidden behind a NUL byte and\n+ limited by the size of a pkt-line (as there will be a single\n+ capability per pkt-line).\n+ * Separate out other information hidden behind NUL bytes (e.g. agent\n+ string as a capability and symrefs can be requested using 'ls-refs')\n+ * Reference advertisement will be omitted unless explicitly requested\n+ * ls-refs command to explicitly request some refs\n+\n+ Detailed Design\n+=================\n+\n+A client can request to speak protocol v2 by sending `version=2` in the\n+side-channel `GIT_PROTOCOL` in the initial request to the server.\n+\n+In protocol v2 communication is command oriented. When first contacting a\n+server a list of capabilities will advertised. Some of these capabilities\n+will be commands which a client can request be executed. Once a command\n+has completed, a client can reuse the connection and request that other\n+commands be executed.\n+\n+ Special Packets\n+-----------------\n+\n+In protocol v2 these special packets will have the following semantics:\n+\n+ * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n+ * '0001' Delimiter Packet (delim-pkt) - separates sections of a message\n+\n+ Capability Advertisement\n+--------------------------\n+\n+A server which decides to communicate (based on a request from a client)\n+using protocol version 2, notifies the client by sending a version string\n+in its initial response followed by an advertisement of its capabilities.\n+Each capability is a key with an optional value. Clients must ignore all\n+unknown keys. Semantics of unknown values are left to the definition of\n+each key. Some capabilities will describe commands which can be requested\n+to be executed by the client.\n+\n+ capability-advertisement = protocol-version\n+\t\t\t capability-list\n+\t\t\t flush-pkt\n+\n+ protocol-version = PKT-LINE(\"version 2\" LF)\n+ capability-list = *capability\n+ capability = PKT-LINE(key[=value] LF)\n+\n+ key = 1*CHAR\n+ value = 1*CHAR\n+ CHAR = 1*(ALPHA / DIGIT / \"-\" / \"_\")\n+\n+A client then responds to select the command it wants with any particular\n+capabilities or arguments. There is then an optional section where the\n+client can provide any command specific parameters or queries.\n+\n+ command-request = command\n+\t\t capability-list\n+\t\t (command-args)\n+\t\t flush-pkt\n+ command = PKT-LINE(\"command=\" key LF)\n+ command-args = delim-pkt\n+\t\t *arg\n+ arg = 1*CHAR\n+\n+The server will then check to ensure that the client's request is\n+comprised of a valid command as well as valid capabilities which were\n+advertised. If the request is valid the server will then execute the\n+command.\n+\n+When a command has finished a client can either request that another\n+command be executed or can terminate the connection by sending an empty\n+request consisting of just a flush-pkt.\n+\n+ Capabilities\n+~~~~~~~~~~~~~~\n+\n+There are two different types of capabilities: normal capabilities,\n+which can be used to to convey information or alter the behavior of a\n+request, and command capabilities, which are the core actions that a\n+client wants to perform (fetch, push, etc).\n+\n+ agent\n+-------\n+\n+The server can advertise the `agent` capability with a value `X` (in the\n+form `agent=X`) to notify the client that the server is running version\n+`X`. The client may optionally send its own agent string by including\n+the `agent` capability with a value `Y` (in the form `agent=Y`) in its\n+request to the server (but it MUST NOT do so if the server did not\n+advertise the agent capability). The `X` and `Y` strings may contain any\n+printable ASCII characters except space (i.e., the byte range 32 < x <\n+127), and are typically of the form \"package/version\" (e.g.,\n+\"git/1.8.3.1\"). The agent strings are purely informative for statistics\n+and debugging purposes, and MUST NOT be used to programmatically assume\n+the presence or absence of particular features.\n+\n+ stateless-rpc\n+---------------\n+\n+If advertised, the `stateless-rpc` capability indicates that the server\n+supports running commands in a stateless-rpc mode, which means that a\n+command lasts for only a single request-response round.\n+\n+Normally a command can last for as many rounds as are required to\n+complete it (multiple for negotiation during fetch or no additional\n+trips in the case of ls-refs). If the client sends the `stateless-rpc`\n+capability with a value of `true` (in the form `stateless-rpc=true`)\n+then the invoked command must only last a single round.\ndiff --git a/Makefile b/Makefile\nindex 3b849c060..18c255428 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -881,6 +881,7 @@ LIB_OBJS += revision.o\n LIB_OBJS += run-command.o\n LIB_OBJS += send-pack.o\n LIB_OBJS += sequencer.o\n+LIB_OBJS += serve.o\n LIB_OBJS += server-info.o\n LIB_OBJS += setup.o\n LIB_OBJS += sha1-array.o\n@@ -1014,6 +1015,7 @@ BUILTIN_OBJS += builtin/rev-parse.o\n BUILTIN_OBJS += builtin/revert.o\n BUILTIN_OBJS += builtin/rm.o\n BUILTIN_OBJS += builtin/send-pack.o\n+BUILTIN_OBJS += builtin/serve.o\n BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-ref.o\ndiff --git a/builtin.h b/builtin.h\nindex f332a1257..3f3fdfc28 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -215,6 +215,7 @@ extern int cmd_rev_parse(int argc, const char **argv, const char *prefix);\n extern int cmd_revert(int argc, const char **argv, const char *prefix);\n extern int cmd_rm(int argc, const char **argv, const char *prefix);\n extern int cmd_send_pack(int argc, const char **argv, const char *prefix);\n+extern int cmd_serve(int argc, const char **argv, const char *prefix);\n extern int cmd_shortlog(int argc, const char **argv, const char *prefix);\n extern int cmd_show(int argc, const char **argv, const char *prefix);\n extern int cmd_show_branch(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/serve.c b/builtin/serve.c\nnew file mode 100644\nindex 000000000..d3fd240bb\n--- /dev/null\n+++ b/builtin/serve.c\n@@ -0,0 +1,30 @@\n+#include \"cache.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"serve.h\"\n+\n+static char const * const serve_usage[] = {\n+\tN_(\"git serve []\"),\n+\tNULL\n+};\n+\n+int cmd_serve(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct serve_options opts = SERVE_OPTIONS_INIT;\n+\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n+\t\t\t N_(\"quit after a single request/response exchange\")),\n+\t\tOPT_BOOL(0, \"advertise-capabilities\", &opts.advertise_capabilities,\n+\t\t\t N_(\"exit immediately after advertising capabilities\")),\n+\t\tOPT_END()\n+\t};\n+\n+\t/* ignore all unknown cmdline switches for now */\n+\targc = parse_options(argc, argv, prefix, options, serve_usage,\n+\t\t\t PARSE_OPT_KEEP_DASHDASH |\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tserve(&opts);\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex f71073dc8..f85d682b6 100644\n--- a/git.c\n+++ b/git.c\n@@ -461,6 +461,7 @@ static struct cmd_struct commands[] = {\n \t{ \"revert\", cmd_revert, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"rm\", cmd_rm, RUN_SETUP },\n \t{ \"send-pack\", cmd_send_pack, RUN_SETUP },\n+\t{ \"serve\", cmd_serve, RUN_SETUP },\n \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n \t{ \"show\", cmd_show, RUN_SETUP },\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\ndiff --git a/serve.c b/serve.c\nnew file mode 100644\nindex 000000000..90e3defe8\n--- /dev/null\n+++ b/serve.c\n@@ -0,0 +1,249 @@\n+#include \"cache.h\"\n+#include \"repository.h\"\n+#include \"config.h\"\n+#include \"pkt-line.h\"\n+#include \"version.h\"\n+#include \"argv-array.h\"\n+#include \"serve.h\"\n+\n+static int always_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\treturn 1;\n+}\n+\n+static int agent_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, git_user_agent_sanitized());\n+\treturn 1;\n+}\n+\n+struct protocol_capability {\n+\t/*\n+\t * The name of the capability. The server uses this name when\n+\t * advertising this capability, and the client uses this name to\n+\t * specify this capability.\n+\t */\n+\tconst char *name;\n+\n+\t/*\n+\t * Function queried to see if a capability should be advertised.\n+\t * Optionally a value can be specified by adding it to 'value'.\n+\t * If a value is added to 'value', the server will advertise this\n+\t * capability as \"=\" instead of \"\".\n+\t */\n+\tint (*advertise)(struct repository *r, struct strbuf *value);\n+\n+\t/*\n+\t * Function called when a client requests the capability as a command.\n+\t * The command request will be provided to the function via 'keys', the\n+\t * capabilities requested, and 'args', the command specific parameters.\n+\t *\n+\t * This field should be NULL for capabilities which are not commands.\n+\t */\n+\tint (*command)(struct repository *r,\n+\t\t struct argv_array *keys,\n+\t\t struct argv_array *args);\n+};\n+\n+static struct protocol_capability capabilities[] = {\n+\t{ \"agent\", agent_advertise, NULL },\n+\t{ \"stateless-rpc\", always_advertise, NULL },\n+};\n+\n+static void advertise_capabilities(void)\n+{\n+\tstruct strbuf capability = STRBUF_INIT;\n+\tstruct strbuf value = STRBUF_INIT;\n+\tint i;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n+\t\tstruct protocol_capability *c = &capabilities[i];\n+\n+\t\tif (c->advertise(the_repository, &value)) {\n+\t\t\tstrbuf_addstr(&capability, c->name);\n+\n+\t\t\tif (value.len) {\n+\t\t\t\tstrbuf_addch(&capability, '=');\n+\t\t\t\tstrbuf_addbuf(&capability, &value);\n+\t\t\t}\n+\n+\t\t\tstrbuf_addch(&capability, '\\n');\n+\t\t\tpacket_write(1, capability.buf, capability.len);\n+\t\t}\n+\n+\t\tstrbuf_reset(&capability);\n+\t\tstrbuf_reset(&value);\n+\t}\n+\n+\tpacket_flush(1);\n+\tstrbuf_release(&capability);\n+\tstrbuf_release(&value);\n+}\n+\n+static struct protocol_capability *get_capability(const char *key)\n+{\n+\tint i;\n+\n+\tif (!key)\n+\t\treturn NULL;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n+\t\tstruct protocol_capability *c = &capabilities[i];\n+\t\tconst char *out;\n+\t\tif (skip_prefix(key, c->name, &out) && (!*out || *out == '='))\n+\t\t\treturn c;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static int is_valid_capability(const char *key)\n+{\n+\tconst struct protocol_capability *c = get_capability(key);\n+\n+\treturn c && c->advertise(the_repository, NULL);\n+}\n+\n+static int is_command(const char *key, struct protocol_capability **command)\n+{\n+\tconst char *out;\n+\n+\tif (skip_prefix(key, \"command=\", &out)) {\n+\t\tstruct protocol_capability *cmd = get_capability(out);\n+\n+\t\tif (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)\n+\t\t\tdie(\"invalid command '%s'\", out);\n+\t\tif (*command)\n+\t\t\tdie(\"command already requested\");\n+\n+\t\t*command = cmd;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int has_capability(const struct argv_array *keys, const char *capability,\n+\t\t const char **value)\n+{\n+\tint i;\n+\tfor (i = 0; i < keys->argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(keys->argv[i], capability, &out) &&\n+\t\t (!*out || *out == '=')) {\n+\t\t\tif (value) {\n+\t\t\t\tif (*out == '=')\n+\t\t\t\t\tout++;\n+\t\t\t\t*value = out;\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+enum request_state {\n+\tPROCESS_REQUEST_KEYS = 0,\n+\tPROCESS_REQUEST_ARGS,\n+\tPROCESS_REQUEST_DONE,\n+};\n+\n+static int process_request(void)\n+{\n+\tenum request_state state = PROCESS_REQUEST_KEYS;\n+\tchar *buffer = packet_buffer;\n+\tunsigned buffer_size = sizeof(packet_buffer);\n+\tint pktlen;\n+\tstruct argv_array keys = ARGV_ARRAY_INIT;\n+\tstruct argv_array args = ARGV_ARRAY_INIT;\n+\tstruct protocol_capability *command = NULL;\n+\n+\twhile (state != PROCESS_REQUEST_DONE) {\n+\t\tswitch (packet_read_with_status(0, NULL, NULL, buffer,\n+\t\t\t\t\t\tbuffer_size, &pktlen,\n+\t\t\t\t\t\tPACKET_READ_CHOMP_NEWLINE)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tBUG(\"Should have already died when seeing EOF\");\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tstate = PROCESS_REQUEST_DONE;\n+\t\t\tcontinue;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tif (state != PROCESS_REQUEST_KEYS)\n+\t\t\t\tdie(\"protocol error\");\n+\t\t\tstate = PROCESS_REQUEST_ARGS;\n+\t\t\t/*\n+\t\t\t * maybe include a check to make sure that a\n+\t\t\t * command/capabilities were given.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (state) {\n+\t\tcase PROCESS_REQUEST_KEYS:\n+\t\t\t/* collect request; a sequence of keys and values */\n+\t\t\tif (is_command(buffer, &command) ||\n+\t\t\t is_valid_capability(buffer))\n+\t\t\t\targv_array_push(&keys, buffer);\n+\t\t\telse\n+\t\t\t\tdie(\"unknown capability '%s'\", buffer);\n+\t\t\tbreak;\n+\t\tcase PROCESS_REQUEST_ARGS:\n+\t\t\t/* collect arguments for the requested command */\n+\t\t\targv_array_push(&args, buffer);\n+\t\t\tbreak;\n+\t\tcase PROCESS_REQUEST_DONE:\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If no command and no keys were given then the client wanted to\n+\t * terminate the connection.\n+\t */\n+\tif (!keys.argc && !args.argc)\n+\t\treturn 1;\n+\n+\tif (!command)\n+\t\tdie(\"no command requested\");\n+\n+\tcommand->command(the_repository, &keys, &args);\n+\n+\targv_array_clear(&keys);\n+\targv_array_clear(&args);\n+\treturn 0;\n+}\n+\n+/* Main serve loop for protocol version 2 */\n+void serve(struct serve_options *options)\n+{\n+\tif (options->advertise_capabilities || !options->stateless_rpc) {\n+\t\t/* serve by default supports v2 */\n+\t\tpacket_write_fmt(1, \"version 2\\n\");\n+\n+\t\tadvertise_capabilities();\n+\t\t/*\n+\t\t * If only the list of capabilities was requested exit\n+\t\t * immediately after advertising capabilities\n+\t\t */\n+\t\tif (options->advertise_capabilities)\n+\t\t\treturn;\n+\t}\n+\n+\t/*\n+\t * If stateless-rpc was requested then exit after\n+\t * a single request/response exchange\n+\t */\n+\tif (options->stateless_rpc) {\n+\t\tprocess_request();\n+\t} else {\n+\t\tfor (;;)\n+\t\t\tif (process_request())\n+\t\t\t\tbreak;\n+\t}\n+}\ndiff --git a/serve.h b/serve.h\nnew file mode 100644\nindex 000000000..fe65ba9f4\n--- /dev/null\n+++ b/serve.h\n@@ -0,0 +1,15 @@\n+#ifndef SERVE_H\n+#define SERVE_H\n+\n+struct argv_array;\n+extern int has_capability(const struct argv_array *keys, const char *capability,\n+\t\t\t const char **value);\n+\n+struct serve_options {\n+\tunsigned advertise_capabilities;\n+\tunsigned stateless_rpc;\n+};\n+#define SERVE_OPTIONS_INIT { 0 }\n+extern void serve(struct serve_options *options);\n+\n+#endif /* SERVE_H */\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nnew file mode 100755\nindex 000000000..b5cc049e5\n--- /dev/null\n+++ b/t/t5701-git-serve.sh\n@@ -0,0 +1,56 @@\n+#!/bin/sh\n+\n+test_description='test git-serve and server commands'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'test capability advertisement' '\n+\tcat >expect <<-EOF &&\n+\tversion 2\n+\tagent=git/$(git version | cut -d\" \" -f3)\n+\tstateless-rpc\n+\t0000\n+\tEOF\n+\n+\tgit serve --advertise-capabilities >out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'stateless-rpc flag does not list capabilities' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\t0000\n+\tEOF\n+\tgit serve --stateless-rpc >out in <<-EOF &&\n+\tfoobar\n+\t0000\n+\tEOF\n+\ttest_must_fail git serve --stateless-rpc 2>err in <<-EOF &&\n+\tagent=git/test\n+\t0000\n+\tEOF\n+\ttest_must_fail git serve --stateless-rpc 2>err in <<-EOF &&\n+\tcommand=foo\n+\tagent=git/test\n+\t0000\n+\tEOF\n+\ttest_must_fail git serve --stateless-rpc 2>err \n---\n builtin/upload-pack.c | 10 ++--\n connect.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++-\n remote.h | 4 ++\n t/t5702-protocol-v2.sh | 28 +++++++++++\n transport.c | 2 +-\n 5 files changed, 160 insertions(+), 7 deletions(-)\n create mode 100755 t/t5702-protocol-v2.sh\n\ndiff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\nindex 8d53e9794..a757df8da 100644\n--- a/builtin/upload-pack.c\n+++ b/builtin/upload-pack.c\n@@ -5,6 +5,7 @@\n #include \"parse-options.h\"\n #include \"protocol.h\"\n #include \"upload-pack.h\"\n+#include \"serve.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -16,6 +17,7 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \tconst char *dir;\n \tint strict = 0;\n \tstruct upload_pack_options opts = { 0 };\n+\tstruct serve_options serve_opts = SERVE_OPTIONS_INIT;\n \tstruct option options[] = {\n \t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n \t\t\t N_(\"quit after a single request/response exchange\")),\n@@ -48,11 +50,9 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \n \tswitch (determine_protocol_version_server()) {\n \tcase protocol_v2:\n-\t\t/*\n-\t\t * fetch support for protocol v2 has not been implemented yet,\n-\t\t * so ignore the request to use v2 and fallback to using v0.\n-\t\t */\n-\t\tupload_pack(&opts);\n+\t\tserve_opts.advertise_capabilities = opts.advertise_refs;\n+\t\tserve_opts.stateless_rpc = opts.stateless_rpc;\n+\t\tserve(&serve_opts);\n \t\tbreak;\n \tcase protocol_v1:\n \t\t/*\ndiff --git a/connect.c b/connect.c\nindex f2157a821..3c653b65b 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -12,9 +12,11 @@\n #include \"sha1-array.h\"\n #include \"transport.h\"\n #include \"strbuf.h\"\n+#include \"version.h\"\n #include \"protocol.h\"\n \n static char *server_capabilities;\n+static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n static const char *parse_feature_value(const char *, const char *, int *);\n \n static int check_ref(const char *name, unsigned int flags)\n@@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n+/* Checks if the server supports the capability 'c' */\n+static int server_supports_v2(const char *c, int die_on_error)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n+\t\t (!*out || *out == '='))\n+\t\t\treturn 1;\n+\t}\n+\n+\tif (die_on_error)\n+\t\tdie(\"server doesn't support '%s'\", c);\n+\n+\treturn 0;\n+}\n+\n+static void process_capabilities_v2(struct packet_reader *reader)\n+{\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL)\n+\t\targv_array_push(&server_capabilities_v2, reader->line);\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\tdie(\"protocol error\");\n+}\n+\n enum protocol_version discover_version(struct packet_reader *reader)\n {\n \tenum protocol_version version = protocol_unknown_version;\n@@ -85,7 +114,7 @@ enum protocol_version discover_version(struct packet_reader *reader)\n \t/* Maybe process capabilities here, at least for v2 */\n \tswitch (version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tprocess_capabilities_v2(reader);\n \t\tbreak;\n \tcase protocol_v1:\n \t\t/* Read the peeked version line */\n@@ -293,6 +322,98 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n \treturn list;\n }\n \n+static int process_ref_v2(const char *line, struct ref ***list)\n+{\n+\tint ret = 1;\n+\tint i = 0;\n+\tstruct object_id old_oid;\n+\tstruct ref *ref;\n+\tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n+\n+\tif (string_list_split(&line_sections, line, ' ', -1) < 2) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tif (get_oid_hex(line_sections.items[i++].string, &old_oid)) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tref = alloc_ref(line_sections.items[i++].string);\n+\n+\toidcpy(&ref->old_oid, &old_oid);\n+\t**list = ref;\n+\t*list = &ref->next;\n+\n+\tfor (; i < line_sections.nr; i++) {\n+\t\tconst char *arg = line_sections.items[i].string;\n+\t\tif (skip_prefix(arg, \"symref-target:\", &arg))\n+\t\t\tref->symref = xstrdup(arg);\n+\n+\t\tif (skip_prefix(arg, \"peeled:\", &arg)) {\n+\t\t\tstruct object_id peeled_oid;\n+\t\t\tchar *peeled_name;\n+\t\t\tstruct ref *peeled;\n+\t\t\tif (get_oid_hex(arg, &peeled_oid)) {\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tpeeled_name = xstrfmt(\"%s^{}\", ref->name);\n+\t\t\tpeeled = alloc_ref(peeled_name);\n+\n+\t\t\toidcpy(&peeled->old_oid, &peeled_oid);\n+\t\t\t**list = peeled;\n+\t\t\t*list = &peeled->next;\n+\n+\t\t\tfree(peeled_name);\n+\t\t}\n+\t}\n+\n+out:\n+\tstring_list_clear(&line_sections, 0);\n+\treturn ret;\n+}\n+\n+struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n+\t\t\t struct ref **list, int for_push,\n+\t\t\t const struct argv_array *ref_patterns)\n+{\n+\tint i;\n+\t*list = NULL;\n+\n+\t/* Check that the server supports the ls-refs command */\n+\t/* Issue request for ls-refs */\n+\tif (server_supports_v2(\"ls-refs\", 1))\n+\t\tpacket_write_fmt(fd_out, \"command=ls-refs\\n\");\n+\n+\tif (server_supports_v2(\"agent\", 0))\n+\t packet_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n+\n+\tpacket_delim(fd_out);\n+\t/* When pushing we don't want to request the peeled tags */\n+\tif (!for_push)\n+\t\tpacket_write_fmt(fd_out, \"peel\\n\");\n+\tpacket_write_fmt(fd_out, \"symrefs\\n\");\n+\tfor (i = 0; ref_patterns && i < ref_patterns->argc; i++) {\n+\t\tpacket_write_fmt(fd_out, \"ref-pattern %s\\n\",\n+\t\t\t\t ref_patterns->argv[i]);\n+\t}\n+\tpacket_flush(fd_out);\n+\n+\t/* Process response from server */\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tif (!process_ref_v2(reader->line, &list))\n+\t\t\tdie(\"invalid ls-refs response: %s\", reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\tdie(\"protocol error\");\n+\n+\treturn list;\n+}\n+\n static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)\n {\n \tint len;\ndiff --git a/remote.h b/remote.h\nindex 2016461df..21d0c776c 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -151,10 +151,14 @@ void free_refs(struct ref *ref);\n \n struct oid_array;\n struct packet_reader;\n+struct argv_array;\n extern struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t\t struct ref **list, unsigned int flags,\n \t\t\t\t struct oid_array *extra_have,\n \t\t\t\t struct oid_array *shallow_points);\n+extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n+\t\t\t\t struct ref **list, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nnew file mode 100755\nindex 000000000..4bf4d61ac\n--- /dev/null\n+++ b/t/t5702-protocol-v2.sh\n@@ -0,0 +1,28 @@\n+#!/bin/sh\n+\n+test_description='test git wire-protocol version 2'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'file://' transport\n+#\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init file_parent &&\n+\ttest_commit -C file_parent one\n+'\n+\n+test_expect_success 'list refs with file:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tls-remote --symref \"file://$(pwd)/file_parent\" >actual 2>log &&\n+\n+\t# Server responded using protocol v2\n+\tcat log &&\n+\tgrep \"git< version 2\" log &&\n+\n+\tgit ls-remote --symref \"file://$(pwd)/file_parent\" >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/transport.c b/transport.c\nindex 83d9dd1df..ffc6b2614 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -204,7 +204,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337433","messageId":"20180125235838.138135-17-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 16/27] transport: convert transport_get_remote_refs to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:27Z","receivedAt":"2018-01-25T23:59:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert 'transport_get_remote_refs()' to optionally take a list of ref\npatterns.\n\nSigned-off-by: Brandon Williams \n---\n builtin/clone.c | 2 +-\n builtin/fetch.c | 4 ++--\n builtin/ls-remote.c | 2 +-\n builtin/remote.c | 2 +-\n transport.c | 7 +++++--\n transport.h | 3 ++-\n 6 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 284651797..6e77d993f 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1121,7 +1121,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (transport->smart_options && !deepen)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n-\trefs = transport_get_remote_refs(transport);\n+\trefs = transport_get_remote_refs(transport, NULL);\n \n \tif (refs) {\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7bbcd26fa..850382f55 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -250,7 +250,7 @@ static void find_non_local_tags(struct transport *transport,\n \tstruct string_list_item *item = NULL;\n \n \tfor_each_ref(add_existing, &existing_refs);\n-\tfor (ref = transport_get_remote_refs(transport); ref; ref = ref->next) {\n+\tfor (ref = transport_get_remote_refs(transport, NULL); ref; ref = ref->next) {\n \t\tif (!starts_with(ref->name, \"refs/tags/\"))\n \t\t\tcontinue;\n \n@@ -336,7 +336,7 @@ static struct ref *get_ref_map(struct transport *transport,\n \t/* opportunistically-updated references: */\n \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n \n-\tconst struct ref *remote_refs = transport_get_remote_refs(transport);\n+\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n \n \tif (refspec_count) {\n \t\tstruct refspec *fetch_refspec;\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex c4be98ab9..c6e9847c5 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -96,7 +96,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (uploadpack != NULL)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n \n-\tref = transport_get_remote_refs(transport);\n+\tref = transport_get_remote_refs(transport, NULL);\n \tif (transport_disconnect(transport))\n \t\treturn 1;\n \ndiff --git a/builtin/remote.c b/builtin/remote.c\nindex d95bf904c..d0b6ff6e2 100644\n--- a/builtin/remote.c\n+++ b/builtin/remote.c\n@@ -862,7 +862,7 @@ static int get_remote_ref_states(const char *name,\n \tif (query) {\n \t\ttransport = transport_get(states->remote, states->remote->url_nr > 0 ?\n \t\t\tstates->remote->url[0] : NULL);\n-\t\tremote_refs = transport_get_remote_refs(transport);\n+\t\tremote_refs = transport_get_remote_refs(transport, NULL);\n \t\ttransport_disconnect(transport);\n \n \t\tstates->queried = 1;\ndiff --git a/transport.c b/transport.c\nindex c54a44630..dfc603b36 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1136,10 +1136,13 @@ int transport_push(struct transport *transport,\n \treturn 1;\n }\n \n-const struct ref *transport_get_remote_refs(struct transport *transport)\n+const struct ref *transport_get_remote_refs(struct transport *transport,\n+\t\t\t\t\t const struct argv_array *ref_patterns)\n {\n \tif (!transport->got_remote_refs) {\n-\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n+\t\ttransport->remote_refs =\n+\t\t\ttransport->vtable->get_refs_list(transport, 0,\n+\t\t\t\t\t\t\t ref_patterns);\n \t\ttransport->got_remote_refs = 1;\n \t}\n \ndiff --git a/transport.h b/transport.h\nindex 731c78b67..4b656f315 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -178,7 +178,8 @@ int transport_push(struct transport *connection,\n \t\t int refspec_nr, const char **refspec, int flags,\n \t\t unsigned int * reject_reasons);\n \n-const struct ref *transport_get_remote_refs(struct transport *transport);\n+const struct ref *transport_get_remote_refs(struct transport *transport,\n+\t\t\t\t\t const struct argv_array *ref_patterns);\n \n int transport_fetch_refs(struct transport *transport, struct ref *refs);\n void transport_unlock_pack(struct transport *transport);\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337434","messageId":"20180125235838.138135-18-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 17/27] ls-remote: pass ref patterns when requesting a remote's refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:28Z","receivedAt":"2018-01-25T23:59:29Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct an argv_array of the ref patterns supplied via the command\nline and pass them to 'transport_get_remote_refs()' to be used when\ncommunicating protocol v2 so that the server can limit the ref\nadvertisement based on the supplied patterns.\n\nSigned-off-by: Brandon Williams \n---\n builtin/ls-remote.c | 7 +++++--\n t/t5702-protocol-v2.sh | 8 ++++++++\n 2 files changed, 13 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex c6e9847c5..caf1051f3 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -43,6 +43,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tint show_symref_target = 0;\n \tconst char *uploadpack = NULL;\n \tconst char **pattern = NULL;\n+\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n \n \tstruct remote *remote;\n \tstruct transport *transport;\n@@ -74,8 +75,10 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (argc > 1) {\n \t\tint i;\n \t\tpattern = xcalloc(argc, sizeof(const char *));\n-\t\tfor (i = 1; i < argc; i++)\n+\t\tfor (i = 1; i < argc; i++) {\n \t\t\tpattern[i - 1] = xstrfmt(\"*/%s\", argv[i]);\n+\t\t\targv_array_push(&ref_patterns, argv[i]);\n+\t\t}\n \t}\n \n \tremote = remote_get(dest);\n@@ -96,7 +99,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (uploadpack != NULL)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n \n-\tref = transport_get_remote_refs(transport, NULL);\n+\tref = transport_get_remote_refs(transport, &ref_patterns);\n \tif (transport_disconnect(transport))\n \t\treturn 1;\n \ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 4bf4d61ac..7d8aeb766 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -25,4 +25,12 @@ test_expect_success 'list refs with file:// using protocol v2' '\n \ttest_cmp actual expect\n '\n \n+test_expect_success 'ref advertisment is filtered with ls-remote using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tls-remote \"file://$(pwd)/file_parent\" master 2>log &&\n+\n+\tgrep \"ref-pattern master\" log &&\n+\t! grep \"refs/tags/\" log\n+'\n+\n test_done\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337435","messageId":"20180125235838.138135-19-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 18/27] fetch: pass ref patterns when fetching","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:29Z","receivedAt":"2018-01-25T23:59:31Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct a list of ref patterns to be passed to\n'transport_get_remote_refs()' from the refspec to be used during the\nfetch. This list of ref patterns will be used to allow the server to\nfilter the ref advertisement when communicating using protocol v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch.c | 12 +++++++++++-\n 1 file changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 850382f55..8128450bf 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -332,11 +332,21 @@ static struct ref *get_ref_map(struct transport *transport,\n \tstruct ref *rm;\n \tstruct ref *ref_map = NULL;\n \tstruct ref **tail = &ref_map;\n+\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n \n \t/* opportunistically-updated references: */\n \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n \n-\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n+\tconst struct ref *remote_refs;\n+\n+\tfor (i = 0; i < refspec_count; i++) {\n+\t\tif (!refspecs[i].exact_sha1)\n+\t\t\targv_array_push(&ref_patterns, refspecs[i].src);\n+\t}\n+\n+\tremote_refs = transport_get_remote_refs(transport, &ref_patterns);\n+\n+\targv_array_clear(&ref_patterns);\n \n \tif (refspec_count) {\n \t\tstruct refspec *fetch_refspec;\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337436","messageId":"20180125235838.138135-20-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 19/27] push: pass ref patterns when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:30Z","receivedAt":"2018-01-25T23:59:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct a list of ref patterns to be passed to 'get_refs_list()' from\nthe refspec to be used during the push. This list of ref patterns will\nbe used to allow the server to filter the ref advertisement when\ncommunicating using protocol v2.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 17 ++++++++++++++++-\n 1 file changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/transport.c b/transport.c\nindex dfc603b36..6ea3905e3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1026,11 +1026,26 @@ int transport_push(struct transport *transport,\n \t\tint porcelain = flags & TRANSPORT_PUSH_PORCELAIN;\n \t\tint pretend = flags & TRANSPORT_PUSH_DRY_RUN;\n \t\tint push_ret, ret, err;\n+\t\tstruct refspec *tmp_rs;\n+\t\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n+\t\tint i;\n \n \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n \t\t\treturn -1;\n \n-\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n+\t\ttmp_rs = parse_push_refspec(refspec_nr, refspec);\n+\t\tfor (i = 0; i < refspec_nr; i++) {\n+\t\t\tif (tmp_rs[i].dst)\n+\t\t\t\targv_array_push(&ref_patterns, tmp_rs[i].dst);\n+\t\t\telse if (tmp_rs[i].src && !tmp_rs[i].exact_sha1)\n+\t\t\t\targv_array_push(&ref_patterns, tmp_rs[i].src);\n+\t\t}\n+\n+\t\tremote_refs = transport->vtable->get_refs_list(transport, 1,\n+\t\t\t\t\t\t\t &ref_patterns);\n+\n+\t\targv_array_clear(&ref_patterns);\n+\t\tfree_refspec(refspec_nr, tmp_rs);\n \n \t\tif (flags & TRANSPORT_PUSH_ALL)\n \t\t\tmatch_flags |= MATCH_REFS_ALL;\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337437","messageId":"20180125235838.138135-14-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 13/27] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:24Z","receivedAt":"2018-01-25T23:59:39Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the ls-refs server command. In protocol v2, the ls-refs\ncommand is used to request the ref advertisement from the server. Since\nit is a command which can be requested (as opposed to mandatory in v1),\na client can sent a number of parameters in its request to limit the ref\nadvertisement based on provided ref-patterns.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/technical/protocol-v2.txt | 32 +++++++++\n Makefile | 1 +\n ls-refs.c | 96 ++++++++++++++++++++++++++\n ls-refs.h | 9 +++\n serve.c | 2 +\n t/t5701-git-serve.sh | 115 ++++++++++++++++++++++++++++++++\n 6 files changed, 255 insertions(+)\n create mode 100644 ls-refs.c\n create mode 100644 ls-refs.h\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 7f619a76c..4683d41ac 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -115,3 +115,35 @@ complete it (multiple for negotiation during fetch or no additional\n trips in the case of ls-refs). If the client sends the `stateless-rpc`\n capability with a value of `true` (in the form `stateless-rpc=true`)\n then the invoked command must only last a single round.\n+\n+ ls-refs\n+---------\n+\n+`ls-refs` is the command used to request a reference advertisement in v2.\n+Unlike the current reference advertisement, ls-refs takes in parameters\n+which can be used to limit the refs sent from the server.\n+\n+Additional features not supported in the base command will be advertised\n+as the value of the command in the capability advertisement in the form\n+of a space separated list of features, e.g. \"=\n+\".\n+\n+ls-refs takes in the following parameters wrapped in packet-lines:\n+\n+ symrefs\n+\tIn addition to the object pointed by it, show the underlying ref\n+\tpointed by it when showing a symbolic ref.\n+ peel\n+\tShow peeled tags.\n+ ref-pattern \n+\tWhen specified, only references matching the one of the provided\n+\tpatterns are displayed.\n+\n+The output of ls-refs is as follows:\n+\n+ output = *ref\n+\t flush-pkt\n+ ref = PKT-LINE(obj-id SP refname *(SP ref-attribute) LF)\n+ ref-attribute = (symref | peeled)\n+ symref = \"symref-target:\" symref-target\n+ peeled = \"peeled:\" obj-id\ndiff --git a/Makefile b/Makefile\nindex 18c255428..e50927cfb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -825,6 +825,7 @@ LIB_OBJS += list-objects-filter-options.o\n LIB_OBJS += ll-merge.o\n LIB_OBJS += lockfile.o\n LIB_OBJS += log-tree.o\n+LIB_OBJS += ls-refs.o\n LIB_OBJS += mailinfo.o\n LIB_OBJS += mailmap.o\n LIB_OBJS += match-trees.o\ndiff --git a/ls-refs.c b/ls-refs.c\nnew file mode 100644\nindex 000000000..70682b4f7\n--- /dev/null\n+++ b/ls-refs.c\n@@ -0,0 +1,96 @@\n+#include \"cache.h\"\n+#include \"repository.h\"\n+#include \"refs.h\"\n+#include \"remote.h\"\n+#include \"argv-array.h\"\n+#include \"ls-refs.h\"\n+#include \"pkt-line.h\"\n+\n+struct ls_refs_data {\n+\tunsigned peel;\n+\tunsigned symrefs;\n+\tstruct argv_array patterns;\n+};\n+\n+/*\n+ * Check if one of the patterns matches the tail part of the ref.\n+ * If no patterns were provided, all refs match.\n+ */\n+static int ref_match(const struct argv_array *patterns, const char *refname)\n+{\n+\tchar *pathbuf;\n+\tint i;\n+\n+\tif (!patterns->argc)\n+\t\treturn 1; /* no restriction */\n+\n+\tpathbuf = xstrfmt(\"/%s\", refname);\n+\tfor (i = 0; i < patterns->argc; i++) {\n+\t\tif (!wildmatch(patterns->argv[i], pathbuf, 0)) {\n+\t\t\tfree(pathbuf);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\tfree(pathbuf);\n+\treturn 0;\n+}\n+\n+static int send_ref(const char *refname, const struct object_id *oid,\n+\t\t int flag, void *cb_data)\n+{\n+\tstruct ls_refs_data *data = cb_data;\n+\tconst char *refname_nons = strip_namespace(refname);\n+\tstruct strbuf refline = STRBUF_INIT;\n+\n+\tif (!ref_match(&data->patterns, refname))\n+\t\treturn 0;\n+\n+\tstrbuf_addf(&refline, \"%s %s\", oid_to_hex(oid), refname_nons);\n+\tif (data->symrefs && flag & REF_ISSYMREF) {\n+\t\tstruct object_id unused;\n+\t\tconst char *symref_target = resolve_ref_unsafe(refname, 0,\n+\t\t\t\t\t\t\t &unused,\n+\t\t\t\t\t\t\t &flag);\n+\n+\t\tif (!symref_target)\n+\t\t\tdie(\"'%s' is a symref but it is not?\", refname);\n+\n+\t\tstrbuf_addf(&refline, \" symref-target:%s\", symref_target);\n+\t}\n+\n+\tif (data->peel) {\n+\t\tstruct object_id peeled;\n+\t\tif (!peel_ref(refname, &peeled))\n+\t\t\tstrbuf_addf(&refline, \" peeled:%s\", oid_to_hex(&peeled));\n+\t}\n+\n+\tstrbuf_addch(&refline, '\\n');\n+\tpacket_write(1, refline.buf, refline.len);\n+\n+\tstrbuf_release(&refline);\n+\treturn 0;\n+}\n+\n+int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)\n+{\n+\tint i;\n+\tstruct ls_refs_data data = { 0, 0, ARGV_ARRAY_INIT };\n+\n+\tfor (i = 0; i < args->argc; i++) {\n+\t\tconst char *arg = args->argv[i];\n+\t\tconst char *out;\n+\n+\t\tif (!strcmp(\"peel\", arg))\n+\t\t\tdata.peel = 1;\n+\t\telse if (!strcmp(\"symrefs\", arg))\n+\t\t\tdata.symrefs = 1;\n+\t\telse if (skip_prefix(arg, \"ref-pattern \", &out))\n+\t\t\targv_array_pushf(&data.patterns, \"*/%s\", out);\n+\t}\n+\n+\thead_ref_namespaced(send_ref, &data);\n+\tfor_each_namespaced_ref(send_ref, &data);\n+\tpacket_flush(1);\n+\targv_array_clear(&data.patterns);\n+\treturn 0;\n+}\ndiff --git a/ls-refs.h b/ls-refs.h\nnew file mode 100644\nindex 000000000..9e4c57bfe\n--- /dev/null\n+++ b/ls-refs.h\n@@ -0,0 +1,9 @@\n+#ifndef LS_REFS_H\n+#define LS_REFS_H\n+\n+struct repository;\n+struct argv_array;\n+extern int ls_refs(struct repository *r, struct argv_array *keys,\n+\t\t struct argv_array *args);\n+\n+#endif /* LS_REFS_H */\ndiff --git a/serve.c b/serve.c\nindex 90e3defe8..2f404154a 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -4,6 +4,7 @@\n #include \"pkt-line.h\"\n #include \"version.h\"\n #include \"argv-array.h\"\n+#include \"ls-refs.h\"\n #include \"serve.h\"\n \n static int always_advertise(struct repository *r,\n@@ -51,6 +52,7 @@ struct protocol_capability {\n static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n \t{ \"stateless-rpc\", always_advertise, NULL },\n+\t{ \"ls-refs\", always_advertise, ls_refs },\n };\n \n static void advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex b5cc049e5..debdc1b8d 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -9,6 +9,7 @@ test_expect_success 'test capability advertisement' '\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tstateless-rpc\n+\tls-refs\n \t0000\n \tEOF\n \n@@ -53,4 +54,118 @@ test_expect_success 'request invalid command' '\n \ttest_i18ngrep \"invalid command\" err\n '\n \n+# Test the basics of ls-refs\n+#\n+test_expect_success 'setup some refs and tags' '\n+\ttest_commit one &&\n+\tgit branch dev master &&\n+\ttest_commit two &&\n+\tgit symbolic-ref refs/heads/release refs/heads/master &&\n+\tgit tag -a -m \"annotated tag\" annotated-tag\n+'\n+\n+test_expect_success 'basics of ls-refs' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse HEAD) HEAD\n+\t$(git rev-parse refs/heads/dev) refs/heads/dev\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/heads/release) refs/heads/release\n+\t$(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag\n+\t$(git rev-parse refs/tags/one) refs/tags/one\n+\t$(git rev-parse refs/tags/two) refs/tags/two\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'basic ref-patterns' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tref-pattern master\n+\tref-pattern one\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/tags/one) refs/tags/one\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'wildcard ref-patterns' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tref-pattern refs/heads/*\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/heads/dev) refs/heads/dev\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/heads/release) refs/heads/release\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'peel parameter' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tpeel\n+\tref-pattern refs/tags/*\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag peeled:$(git rev-parse refs/tags/annotated-tag^{})\n+\t$(git rev-parse refs/tags/one) refs/tags/one\n+\t$(git rev-parse refs/tags/two) refs/tags/two\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'symrefs parameter' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tsymrefs\n+\tref-pattern refs/heads/*\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/heads/dev) refs/heads/dev\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/heads/release) refs/heads/release symref-target:refs/heads/master\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n test_done\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337438","messageId":"20180125235838.138135-23-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 22/27] transport-helper: remove name parameter","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:33Z","receivedAt":"2018-01-25T23:59:41Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Commit 266f1fdfa (transport-helper: be quiet on read errors from\nhelpers, 2013-06-21) removed a call to 'die()' which printed the name of\nthe remote helper passed in to the 'recvline_fh()' function using the\n'name' parameter. Once the call to 'die()' was removed the parameter\nwas no longer necessary but wasn't removed. Clean up 'recvline_fh()'\nparameter list by removing the 'name' parameter.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 4c334b5ee..d72155768 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -49,7 +49,7 @@ static void sendline(struct helper_data *helper, struct strbuf *buffer)\n \t\tdie_errno(\"Full write to remote helper failed\");\n }\n \n-static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)\n+static int recvline_fh(FILE *helper, struct strbuf *buffer)\n {\n \tstrbuf_reset(buffer);\n \tif (debug)\n@@ -67,7 +67,7 @@ static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)\n \n static int recvline(struct helper_data *helper, struct strbuf *buffer)\n {\n-\treturn recvline_fh(helper->out, buffer, helper->name);\n+\treturn recvline_fh(helper->out, buffer);\n }\n \n static void write_constant(int fd, const char *str)\n@@ -586,7 +586,7 @@ static int process_connect_service(struct transport *transport,\n \t\tgoto exit;\n \n \tsendline(data, &cmdbuf);\n-\tif (recvline_fh(input, &cmdbuf, name))\n+\tif (recvline_fh(input, &cmdbuf))\n \t\texit(128);\n \n \tif (!strcmp(cmdbuf.buf, \"\")) {\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337439","messageId":"20180125235838.138135-24-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 23/27] transport-helper: refactor process_connect_service","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:34Z","receivedAt":"2018-01-25T23:59:44Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"A future patch will need to take advantage of the logic which runs and\nprocesses the response of the connect command on a remote helper so\nfactor out this logic from 'process_connect_service()' and place it into\na helper function 'run_connect()'.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 67 +++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 38 insertions(+), 29 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex d72155768..c032a2a87 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -545,14 +545,13 @@ static int fetch_with_import(struct transport *transport,\n \treturn 0;\n }\n \n-static int process_connect_service(struct transport *transport,\n-\t\t\t\t const char *name, const char *exec)\n+static int run_connect(struct transport *transport, struct strbuf *cmdbuf)\n {\n \tstruct helper_data *data = transport->data;\n-\tstruct strbuf cmdbuf = STRBUF_INIT;\n-\tstruct child_process *helper;\n-\tint r, duped, ret = 0;\n+\tint ret = 0;\n+\tint duped;\n \tFILE *input;\n+\tstruct child_process *helper;\n \n \thelper = get_helper(transport);\n \n@@ -568,44 +567,54 @@ static int process_connect_service(struct transport *transport,\n \tinput = xfdopen(duped, \"r\");\n \tsetvbuf(input, NULL, _IONBF, 0);\n \n+\tsendline(data, cmdbuf);\n+\tif (recvline_fh(input, cmdbuf))\n+\t\texit(128);\n+\n+\tif (!strcmp(cmdbuf->buf, \"\")) {\n+\t\tdata->no_disconnect_req = 1;\n+\t\tif (debug)\n+\t\t\tfprintf(stderr, \"Debug: Smart transport connection \"\n+\t\t\t\t\"ready.\\n\");\n+\t\tret = 1;\n+\t} else if (!strcmp(cmdbuf->buf, \"fallback\")) {\n+\t\tif (debug)\n+\t\t\tfprintf(stderr, \"Debug: Falling back to dumb \"\n+\t\t\t\t\"transport.\\n\");\n+\t} else {\n+\t\tdie(\"Unknown response to connect: %s\",\n+\t\t\tcmdbuf->buf);\n+\t}\n+\n+\tfclose(input);\n+\treturn ret;\n+}\n+\n+static int process_connect_service(struct transport *transport,\n+\t\t\t\t const char *name, const char *exec)\n+{\n+\tstruct helper_data *data = transport->data;\n+\tstruct strbuf cmdbuf = STRBUF_INIT;\n+\tint ret = 0;\n+\n \t/*\n \t * Handle --upload-pack and friends. This is fire and forget...\n \t * just warn if it fails.\n \t */\n \tif (strcmp(name, exec)) {\n-\t\tr = set_helper_option(transport, \"servpath\", exec);\n+\t\tint r = set_helper_option(transport, \"servpath\", exec);\n \t\tif (r > 0)\n \t\t\twarning(\"Setting remote service path not supported by protocol.\");\n \t\telse if (r < 0)\n \t\t\twarning(\"Invalid remote service path.\");\n \t}\n \n-\tif (data->connect)\n+\tif (data->connect) {\n \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n-\telse\n-\t\tgoto exit;\n-\n-\tsendline(data, &cmdbuf);\n-\tif (recvline_fh(input, &cmdbuf))\n-\t\texit(128);\n-\n-\tif (!strcmp(cmdbuf.buf, \"\")) {\n-\t\tdata->no_disconnect_req = 1;\n-\t\tif (debug)\n-\t\t\tfprintf(stderr, \"Debug: Smart transport connection \"\n-\t\t\t\t\"ready.\\n\");\n-\t\tret = 1;\n-\t} else if (!strcmp(cmdbuf.buf, \"fallback\")) {\n-\t\tif (debug)\n-\t\t\tfprintf(stderr, \"Debug: Falling back to dumb \"\n-\t\t\t\t\"transport.\\n\");\n-\t} else\n-\t\tdie(\"Unknown response to connect: %s\",\n-\t\t\tcmdbuf.buf);\n+\t\tret = run_connect(transport, &cmdbuf);\n+\t}\n \n-exit:\n \tstrbuf_release(&cmdbuf);\n-\tfclose(input);\n \treturn ret;\n }\n \n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337440","messageId":"20180125235838.138135-16-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 15/27] transport: convert get_refs_list to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:26Z","receivedAt":"2018-01-25T23:59:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert the 'struct transport' virtual function 'get_refs_list()' to\noptionally take an argv_array of ref patterns. When communicating with\na server using protocol v2 these ref patterns can be sent when\nrequesting a listing of their refs allowing the server to filter the\nrefs it sends based on the sent patterns.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 5 +++--\n transport-internal.h | 4 +++-\n transport.c | 16 +++++++++-------\n 3 files changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 508015023..4c334b5ee 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1026,7 +1026,8 @@ static int has_attribute(const char *attrs, const char *attr) {\n \t}\n }\n \n-static struct ref *get_refs_list(struct transport *transport, int for_push)\n+static struct ref *get_refs_list(struct transport *transport, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns)\n {\n \tstruct helper_data *data = transport->data;\n \tstruct child_process *helper;\n@@ -1039,7 +1040,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)\n \n \tif (process_connect(transport, for_push)) {\n \t\tdo_take_over(transport);\n-\t\treturn transport->vtable->get_refs_list(transport, for_push);\n+\t\treturn transport->vtable->get_refs_list(transport, for_push, ref_patterns);\n \t}\n \n \tif (data->push && for_push)\ndiff --git a/transport-internal.h b/transport-internal.h\nindex 3c1a29d72..a67657ce3 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -3,6 +3,7 @@\n \n struct ref;\n struct transport;\n+struct argv_array;\n \n struct transport_vtable {\n \t/**\n@@ -21,7 +22,8 @@ struct transport_vtable {\n \t * the ref without a huge amount of effort, it should store it\n \t * in the ref's old_sha1 field; otherwise it should be all 0.\n \t **/\n-\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push);\n+\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns);\n \n \t/**\n \t * Fetch the objects for the given refs. Note that this gets\ndiff --git a/transport.c b/transport.c\nindex ffc6b2614..c54a44630 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -72,7 +72,7 @@ struct bundle_transport_data {\n \tstruct bundle_header header;\n };\n \n-static struct ref *get_refs_from_bundle(struct transport *transport, int for_push)\n+static struct ref *get_refs_from_bundle(struct transport *transport, int for_push, const struct argv_array *ref_patterns)\n {\n \tstruct bundle_transport_data *data = transport->data;\n \tstruct ref *result = NULL;\n@@ -189,7 +189,8 @@ static int connect_setup(struct transport *transport, int for_push)\n \treturn 0;\n }\n \n-static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n+static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n+\t\t\t\t\tconst struct argv_array *ref_patterns)\n {\n \tstruct git_transport_data *data = transport->data;\n \tstruct ref *refs = NULL;\n@@ -204,7 +205,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);\n+\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push,\n+\t\t\t\tref_patterns);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n@@ -250,7 +252,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.update_shallow = data->options.update_shallow;\n \n \tif (!data->got_remote_heads)\n-\t\trefs_tmp = get_refs_via_connect(transport, 0);\n+\t\trefs_tmp = get_refs_via_connect(transport, 0, NULL);\n \n \tswitch (data->version) {\n \tcase protocol_v2:\n@@ -568,7 +570,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tint ret = 0;\n \n \tif (!data->got_remote_heads)\n-\t\tget_refs_via_connect(transport, 1);\n+\t\tget_refs_via_connect(transport, 1, NULL);\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n@@ -1028,7 +1030,7 @@ int transport_push(struct transport *transport,\n \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n \t\t\treturn -1;\n \n-\t\tremote_refs = transport->vtable->get_refs_list(transport, 1);\n+\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n \n \t\tif (flags & TRANSPORT_PUSH_ALL)\n \t\t\tmatch_flags |= MATCH_REFS_ALL;\n@@ -1137,7 +1139,7 @@ int transport_push(struct transport *transport,\n const struct ref *transport_get_remote_refs(struct transport *transport)\n {\n \tif (!transport->got_remote_refs) {\n-\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0);\n+\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n \t\ttransport->got_remote_refs = 1;\n \t}\n \n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337441","messageId":"20180125235838.138135-25-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 24/27] transport-helper: introduce stateless-connect","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:35Z","receivedAt":"2018-01-25T23:59:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the transport-helper capability 'stateless-connect'. This\ncapability indicates that the transport-helper can be requested to run\nthe 'stateless-connect' command which should attempt to make a\nstateless connection with a remote end. Once established, the\nconnection can be used by the git client to communicate with\nthe remote end natively in a stateless-rpc manner as supported by\nprotocol v2. This means that the client must send everything the server\nneeds in a single request as the client must not assume any\nstate-storing on the part of the server or transport.\n\nIf a stateless connection cannot be established then the remote-helper\nwill respond in the same manner as the 'connect' command indicating that\nthe client should fallback to using the dumb remote-helper commands.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 8 ++++++++\n transport.c | 1 +\n transport.h | 6 ++++++\n 3 files changed, 15 insertions(+)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c032a2a87..82eb57c4a 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -26,6 +26,7 @@ struct helper_data {\n \t\toption : 1,\n \t\tpush : 1,\n \t\tconnect : 1,\n+\t\tstateless_connect : 1,\n \t\tsigned_tags : 1,\n \t\tcheck_connectivity : 1,\n \t\tno_disconnect_req : 1,\n@@ -188,6 +189,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\trefspecs[refspec_nr++] = xstrdup(arg);\n \t\t} else if (!strcmp(capname, \"connect\")) {\n \t\t\tdata->connect = 1;\n+\t\t} else if (!strcmp(capname, \"stateless-connect\")) {\n+\t\t\tdata->stateless_connect = 1;\n \t\t} else if (!strcmp(capname, \"signed-tags\")) {\n \t\t\tdata->signed_tags = 1;\n \t\t} else if (skip_prefix(capname, \"export-marks \", &arg)) {\n@@ -612,6 +615,11 @@ static int process_connect_service(struct transport *transport,\n \tif (data->connect) {\n \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n \t\tret = run_connect(transport, &cmdbuf);\n+\t} else if (data->stateless_connect) {\n+\t\tstrbuf_addf(&cmdbuf, \"stateless-connect %s\\n\", name);\n+\t\tret = run_connect(transport, &cmdbuf);\n+\t\tif (ret)\n+\t\t\ttransport->stateless_rpc = 1;\n \t}\n \n \tstrbuf_release(&cmdbuf);\ndiff --git a/transport.c b/transport.c\nindex 4fdbd9adc..aafb8fbb4 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -250,6 +250,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n+\targs.stateless_rpc = transport->stateless_rpc;\n \n \tif (!data->got_remote_heads)\n \t\trefs_tmp = get_refs_via_connect(transport, 0, NULL);\ndiff --git a/transport.h b/transport.h\nindex 4b656f315..9eac809ee 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -55,6 +55,12 @@ struct transport {\n \t */\n \tunsigned cloning : 1;\n \n+\t/*\n+\t * Indicates that the transport is connected via a half-duplex\n+\t * connection and should operate in stateless-rpc mode.\n+\t */\n+\tunsigned stateless_rpc : 1;\n+\n \t/*\n \t * These strings will be passed to the {pre, post}-receive hook,\n \t * on the remote side, if both sides support the push options capability.\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337442","messageId":"20180125235838.138135-27-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 26/27] remote-curl: create copy of the service name","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:37Z","receivedAt":"2018-01-25T23:59:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Make a copy of the service name being requested instead of relying on\nthe buffer pointed to by the passed in 'const char *' to remain\nunchanged.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex dae8a4a48..4086aa733 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -165,7 +165,7 @@ static int set_option(const char *name, const char *value)\n }\n \n struct discovery {\n-\tconst char *service;\n+\tchar *service;\n \tchar *buf_alloc;\n \tchar *buf;\n \tsize_t len;\n@@ -257,6 +257,7 @@ static void free_discovery(struct discovery *d)\n \t\tfree(d->shallow.oid);\n \t\tfree(d->buf_alloc);\n \t\tfree_refs(d->refs);\n+\t\tfree(d->service);\n \t\tfree(d);\n \t}\n }\n@@ -343,7 +344,7 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \t\twarning(_(\"redirecting to %s\"), url.buf);\n \n \tlast= xcalloc(1, sizeof(*last_discovery));\n-\tlast->service = service;\n+\tlast->service = xstrdup(service);\n \tlast->buf_alloc = strbuf_detach(&buffer, &last->len);\n \tlast->buf = last->buf_alloc;\n \n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337443","messageId":"20180125235838.138135-28-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 27/27] remote-curl: implement stateless-connect command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:38Z","receivedAt":"2018-01-25T23:59:55Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach remote-curl the 'stateless-connect' command which is used to\nestablish a stateless connection with servers which support protocol\nversion 2. This allows remote-curl to act as a proxy, allowing the git\nclient to communicate natively with a remote end, simply using\nremote-curl as a pass through to convert requests to http.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 185 ++++++++++++++++++++++++++++++++++++++++++++++++-\n t/t5702-protocol-v2.sh | 41 +++++++++++\n 2 files changed, 224 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 4086aa733..a17c7e228 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -171,6 +171,7 @@ struct discovery {\n \tsize_t len;\n \tstruct ref *refs;\n \tstruct oid_array shallow;\n+\tenum protocol_version version;\n \tunsigned proto_git : 1;\n };\n static struct discovery *last_discovery;\n@@ -184,9 +185,13 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n-\tswitch (discover_version(&reader)) {\n+\theads->version = discover_version(&reader);\n+\tswitch (heads->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\t/*\n+\t\t * Do nothing. Client should run 'stateless-connect' and\n+\t\t * request the refs themselves.\n+\t\t */\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n@@ -1047,6 +1052,178 @@ static void parse_push(struct strbuf *buf)\n \tfree(specs);\n }\n \n+struct proxy_state {\n+\tchar *service_name;\n+\tchar *service_url;\n+\tstruct curl_slist *headers;\n+\tstruct strbuf request_buffer;\n+\tint in;\n+\tint out;\n+\tstruct packet_reader reader;\n+\tsize_t pos;\n+\tint seen_flush;\n+};\n+\n+static void proxy_state_init(struct proxy_state *p, const char *service_name)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tmemset(p, 0, sizeof(*p));\n+\tp->service_name = xstrdup(service_name);\n+\n+\tp->in = 0;\n+\tp->out = 1;\n+\tstrbuf_init(&p->request_buffer, 0);\n+\n+\tstrbuf_addf(&buf, \"%s%s\", url.buf, p->service_name);\n+\tp->service_url = strbuf_detach(&buf, NULL);\n+\n+\tp->headers = http_copy_default_headers();\n+\n+\tstrbuf_addf(&buf, \"Content-Type: application/x-%s-request\", p->service_name);\n+\tp->headers = curl_slist_append(p->headers, buf.buf);\n+\tstrbuf_reset(&buf);\n+\n+\tstrbuf_addf(&buf, \"Accept: application/x-%s-result\", p->service_name);\n+\tp->headers = curl_slist_append(p->headers, buf.buf);\n+\n+\tp->headers = curl_slist_append(p->headers, \"Transfer-Encoding: chunked\");\n+\n+\tpacket_reader_init(&p->reader, p->in, NULL, 0,\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n+static void proxy_state_clear(struct proxy_state *p)\n+{\n+\tfree(p->service_name);\n+\tfree(p->service_url);\n+\tcurl_slist_free_all(p->headers);\n+\tstrbuf_release(&p->request_buffer);\n+}\n+\n+static size_t proxy_in(char *buffer, size_t eltsize,\n+\t\t size_t nmemb, void *userdata)\n+{\n+\tsize_t max = eltsize * nmemb;\n+\tstruct proxy_state *p = userdata;\n+\tsize_t avail = p->request_buffer.len - p->pos;\n+\n+\tif (!avail) {\n+\t\tif (p->seen_flush) {\n+\t\t\tp->seen_flush = 0;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tstrbuf_reset(&p->request_buffer);\n+\t\tswitch (packet_reader_read(&p->reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tdie(\"unexpected EOF when reading from parent process\");\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tpacket_buf_write_len(&p->request_buffer, p->reader.line,\n+\t\t\t\t\t p->reader.pktlen);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tpacket_buf_delim(&p->request_buffer);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tpacket_buf_flush(&p->request_buffer);\n+\t\t\tp->seen_flush = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tp->pos = 0;\n+\t\tavail = p->request_buffer.len;\n+\t}\n+\n+\tif (max < avail)\n+\t\tavail = max;\n+\tmemcpy(buffer, p->request_buffer.buf + p->pos, avail);\n+\tp->pos += avail;\n+\treturn avail;\n+}\n+\n+static size_t proxy_out(char *buffer, size_t eltsize,\n+\t\t\tsize_t nmemb, void *userdata)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct proxy_state *p = userdata;\n+\n+\twrite_or_die(p->out, buffer, size);\n+\treturn size;\n+}\n+\n+static int proxy_post(struct proxy_state *p)\n+{\n+\tstruct active_request_slot *slot;\n+\tint err;\n+\n+\tslot = get_active_slot();\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, p->headers);\n+\n+\t/* Setup function to read request from client */\n+\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_READDATA, p);\n+\n+\t/* Setup function to write server response to client */\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);\n+\n+\terr = run_slot(slot, NULL);\n+\n+\tif (err != HTTP_OK)\n+\t\terr = -1;\n+\n+\treturn err;\n+}\n+\n+static int stateless_connect(const char *service_name)\n+{\n+\tstruct discovery *discover;\n+\tstruct proxy_state p;\n+\n+\t/*\n+\t * Run the info/refs request and see if the server supports protocol\n+\t * v2. If and only if the server supports v2 can we successfully\n+\t * establish a stateless connection, otherwise we need to tell the\n+\t * client to fallback to using other transport helper functions to\n+\t * complete their request.\n+\t */\n+\tdiscover = discover_refs(service_name, 0);\n+\tif (discover->version != protocol_v2) {\n+\t\tprintf(\"fallback\\n\");\n+\t\tfflush(stdout);\n+\t\treturn -1;\n+\t} else {\n+\t\t/* Stateless Connection established */\n+\t\tprintf(\"\\n\");\n+\t\tfflush(stdout);\n+\t}\n+\n+\tproxy_state_init(&p, service_name);\n+\n+\t/*\n+\t * Dump the capability listing that we got from the server earlier\n+\t * during the info/refs request.\n+\t */\n+\twrite_or_die(p.out, discover->buf, discover->len);\n+\n+\t/* Peek the next packet line. Until we see EOF keep sending POSTs */\n+\twhile (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {\n+\t\tif (proxy_post(&p)) {\n+\t\t\t/* We would have an err here */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tproxy_state_clear(&p);\n+\treturn 0;\n+}\n+\n int cmd_main(int argc, const char **argv)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -1115,12 +1292,16 @@ int cmd_main(int argc, const char **argv)\n \t\t\tfflush(stdout);\n \n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n+\t\t\tprintf(\"stateless-connect\\n\");\n \t\t\tprintf(\"fetch\\n\");\n \t\t\tprintf(\"option\\n\");\n \t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"check-connectivity\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n+\t\t} else if (skip_prefix(buf.buf, \"stateless-connect \", &arg)) {\n+\t\t\tif (!stateless_connect(arg))\n+\t\t\t\tbreak;\n \t\t} else {\n \t\t\terror(\"remote-curl: unknown command '%s' from git\", buf.buf);\n \t\t\treturn 1;\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 3e411e178..ada69ac09 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -73,4 +73,45 @@ test_expect_success 'ref advertisment is filtered during fetch using protocol v2\n \t! grep \"refs/tags/\" log\n '\n \n+# Test protocol v2 with 'http://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one\n+'\n+\n+test_expect_success 'clone with http:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 GIT_TRACE_CURL=1 git -c protocol.version=2 \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"Git-Protocol: version=2\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log\n+'\n+\n+test_expect_success 'fetch with http:// using protocol v2' '\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=2 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337444","messageId":"20180125235838.138135-26-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 25/27] pkt-line: add packet_buf_write_len function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:36Z","receivedAt":"2018-01-26T00:00:00Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add the 'packet_buf_write_len()' function which allows for writing an\narbitrary length buffer into a 'struct strbuf' and formatting it in\npacket-line format.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 16 ++++++++++++++++\n pkt-line.h | 1 +\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 726e109ca..5a8a17ecc 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -215,6 +215,22 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n \tva_end(args);\n }\n \n+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len)\n+{\n+\tsize_t orig_len, n;\n+\n+\torig_len = buf->len;\n+\tstrbuf_addstr(buf, \"0000\");\n+\tstrbuf_add(buf, data, len);\n+\tn = buf->len - orig_len;\n+\n+\tif (n > LARGE_PACKET_MAX)\n+\t\tdie(\"protocol error: impossibly long line\");\n+\n+\tset_packet_header(&buf->buf[orig_len], n);\n+\tpacket_trace(buf->buf + orig_len + 4, n - 4, 1);\n+}\n+\n int write_packetized_from_fd(int fd_in, int fd_out)\n {\n \tstatic char buf[LARGE_PACKET_DATA_MAX];\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 16fe8bdbf..63724d4bf 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -26,6 +26,7 @@ void packet_buf_flush(struct strbuf *buf);\n void packet_buf_delim(struct strbuf *buf);\n void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len);\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int write_packetized_from_fd(int fd_in, int fd_out);\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337445","messageId":"20180125235838.138135-21-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 20/27] upload-pack: introduce fetch server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:31Z","receivedAt":"2018-01-26T00:00:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the 'fetch' server command.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/technical/protocol-v2.txt | 121 +++++++++++++\n serve.c | 2 +\n t/t5701-git-serve.sh | 1 +\n upload-pack.c | 293 ++++++++++++++++++++++++++++++++\n upload-pack.h | 5 +\n 5 files changed, 422 insertions(+)\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 4683d41ac..ca09a2cfe 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -147,3 +147,124 @@ The output of ls-refs is as follows:\n ref-attribute = (symref | peeled)\n symref = \"symref-target:\" symref-target\n peeled = \"peeled:\" obj-id\n+\n+ fetch\n+-------\n+\n+`fetch` is the command used to fetch a packfile in v2. It can be looked\n+at as a modified version of the v1 fetch where the ref-advertisement is\n+stripped out (since the `ls-refs` command fills that role) and the\n+message format is tweaked to eliminate redundancies and permit easy\n+addition of future extensions.\n+\n+Additional features not supported in the base command will be advertised\n+as the value of the command in the capability advertisement in the form\n+of a space separated list of features, e.g. \"=\n+\".\n+\n+A `fetch` request can take the following parameters wrapped in\n+packet-lines:\n+\n+ want \n+\tIndicates to the server an object which the client wants to\n+\tretrieve.\n+\n+ have \n+\tIndicates to the server an object which the client has locally.\n+\tThis allows the server to make a packfile which only contains\n+\tthe objects that the client needs. Multiple 'have' lines can be\n+\tsupplied.\n+\n+ done\n+\tIndicates to the server that negotiation should terminate (or\n+\tnot even begin if performing a clone) and that the server should\n+\tuse the information supplied in the request to construct the\n+\tpackfile.\n+\n+ thin-pack\n+\tRequest that a thin pack be sent, which is a pack with deltas\n+\twhich reference base objects not contained within the pack (but\n+\tare known to exist at the receiving end). This can reduce the\n+\tnetwork traffic significantly, but it requires the receiving end\n+\tto know how to \"thicken\" these packs by adding the missing bases\n+\tto the pack.\n+\n+ no-progress\n+\tRequest that progress information that would normally be sent on\n+\tside-band channel 2, during the packfile transfer, should not be\n+\tsent. However, the side-band channel 3 is still used for error\n+\tresponses.\n+\n+ include-tag\n+\tRequest that annotated tags should be sent if the objects they\n+\tpoint to are being sent.\n+\n+ ofs-delta\n+\tIndicate that the client understands PACKv2 with delta referring\n+\tto its base by position in pack rather than by an oid. That is,\n+\tthey can read OBJ_OFS_DELTA (ake type 6) in a packfile.\n+\n+The response of `fetch` is broken into a number of sections separated by\n+delimiter packets (0001), with each section beginning with its section\n+header.\n+\n+ output = *section\n+ section = (acknowledgments | packfile)\n+\t (flush-pkt | delim-pkt)\n+\n+ acknowledgments = PKT-LINE(\"acknowledgments\" LF)\n+\t\t *(ready | nak | ack)\n+ ready = PKT-LINE(\"ready\" LF)\n+ nak = PKT-LINE(\"NAK\" LF)\n+ ack = PKT-LINE(\"ACK\" SP obj-id LF)\n+\n+ packfile = PKT-LINE(\"packfile\" LF)\n+\t [PACKFILE]\n+\n+----\n+ acknowledgments section\n+\t* Always begins with the section header \"acknowledgments\"\n+\n+\t* The server will respond with \"NAK\" if none of the object ids sent\n+\t as have lines were common.\n+\n+\t* The server will respond with \"ACK obj-id\" for all of the\n+\t object ids sent as have lines which are common.\n+\n+\t* A response cannot have both \"ACK\" lines as well as a \"NAK\"\n+\t line.\n+\n+\t* The server will respond with a \"ready\" line indicating that\n+\t the server has found an acceptable common base and is ready to\n+\t make and send a packfile (which will be found in the packfile\n+\t section of the same response)\n+\n+\t* If the client determines that it is finished with negotiations\n+\t by sending a \"done\" line, the acknowledgments sections can be\n+\t omitted from the server's response as an optimization.\n+\n+\t* If the server has found a suitable cut point and has decided\n+\t to send a \"ready\" line, then the server can decide to (as an\n+\t optimization) omit any \"ACK\" lines it would have sent during\n+\t its response. This is because the server will have already\n+\t determined the objects it plans to send to the client and no\n+\t further negotiation is needed.\n+\n+----\n+ packfile section\n+\t* Always begins with the section header \"packfile\"\n+\n+\t* The transmission of the packfile begins immediately after the\n+\t section header\n+\n+\t* The data transfer of the packfile is always multiplexed, using\n+\t the same semantics of the 'side-band-64k' capability from\n+\t protocol version 1. This means that each packet, during the\n+\t packfile data stream, is made up of a leading 4-byte pkt-line\n+\t length (typical of the pkt-line format), followed by a 1-byte\n+\t stream code, followed by the actual data.\n+\n+\t The stream code can be one of:\n+\t\t1 - pack data\n+\t\t2 - progress messages\n+\t\t3 - fatal error message just before stream aborts\ndiff --git a/serve.c b/serve.c\nindex 2f404154a..e0235e2bc 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -6,6 +6,7 @@\n #include \"argv-array.h\"\n #include \"ls-refs.h\"\n #include \"serve.h\"\n+#include \"upload-pack.h\"\n \n static int always_advertise(struct repository *r,\n \t\t\t struct strbuf *value)\n@@ -53,6 +54,7 @@ static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n \t{ \"stateless-rpc\", always_advertise, NULL },\n \t{ \"ls-refs\", always_advertise, ls_refs },\n+\t{ \"fetch\", always_advertise, upload_pack_v2 },\n };\n \n static void advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex debdc1b8d..e3bc08667 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -10,6 +10,7 @@ test_expect_success 'test capability advertisement' '\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tstateless-rpc\n \tls-refs\n+\tfetch\n \t0000\n \tEOF\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex 42d83d5b1..f7944ffdc 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -18,6 +18,7 @@\n #include \"prio-queue.h\"\n #include \"protocol.h\"\n #include \"upload-pack.h\"\n+#include \"serve.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -1065,3 +1066,295 @@ void upload_pack(struct upload_pack_options *options)\n \t\tcreate_pack_file();\n \t}\n }\n+\n+struct upload_pack_data {\n+\tstruct object_array wants;\n+\tstruct oid_array haves;\n+\n+\tunsigned stateless_rpc : 1;\n+\n+\tunsigned use_thin_pack : 1;\n+\tunsigned use_ofs_delta : 1;\n+\tunsigned no_progress : 1;\n+\tunsigned use_include_tag : 1;\n+\tunsigned done : 1;\n+};\n+\n+#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }\n+\n+static void upload_pack_data_clear(struct upload_pack_data *data)\n+{\n+\tobject_array_clear(&data->wants);\n+\toid_array_clear(&data->haves);\n+}\n+\n+static int parse_want(const char *line)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"want \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tstruct object *o;\n+\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"git upload-pack: protocol error, \"\n+\t\t\t \"expected to get oid, not '%s'\", line);\n+\n+\t\to = parse_object(&oid);\n+\t\tif (!o) {\n+\t\t\tpacket_write_fmt(1,\n+\t\t\t\t\t \"ERR upload-pack: not our ref %s\",\n+\t\t\t\t\t oid_to_hex(&oid));\n+\t\t\tdie(\"git upload-pack: not our ref %s\",\n+\t\t\t oid_to_hex(&oid));\n+\t\t}\n+\n+\t\tif (!(o->flags & WANTED)) {\n+\t\t\to->flags |= WANTED;\n+\t\t\tadd_object_array(o, NULL, &want_obj);\n+\t\t}\n+\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int parse_have(const char *line, struct oid_array *haves)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"have \", &arg)) {\n+\t\tstruct object_id oid;\n+\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"git upload-pack: expected SHA1 object, got '%s'\", arg);\n+\t\toid_array_append(haves, &oid);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void process_args(struct argv_array *args, struct upload_pack_data *data)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < args->argc; i++) {\n+\t\tconst char *arg = args->argv[i];\n+\n+\t\t/* process want */\n+\t\tif (parse_want(arg))\n+\t\t\tcontinue;\n+\t\t/* process have line */\n+\t\tif (parse_have(arg, &data->haves))\n+\t\t\tcontinue;\n+\n+\t\t/* process args like thin-pack */\n+\t\tif (!strcmp(arg, \"thin-pack\")) {\n+\t\t\tuse_thin_pack = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"ofs-delta\")) {\n+\t\t\tuse_ofs_delta = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"no-progress\")) {\n+\t\t\tno_progress = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"include-tag\")) {\n+\t\t\tuse_include_tag = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"done\")) {\n+\t\t\tdata->done = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* ignore unknown lines maybe? */\n+\t\tdie(\"unexpect line: '%s'\", arg);\n+\t}\n+}\n+\n+static void read_haves(struct upload_pack_data *data)\n+{\n+\tstruct packet_reader reader;\n+\tpacket_reader_init(&reader, 0, NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n+\n+\t\tif (parse_have(reader.line, &data->haves))\n+\t\t\tcontinue;\n+\t\tif (!strcmp(reader.line, \"done\")) {\n+\t\t\tdata->done = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\tif (reader.status != PACKET_READ_FLUSH)\n+\t\tdie(\"ERROR\");\n+}\n+\n+static int process_haves(struct oid_array *haves, struct oid_array *common)\n+{\n+\tint i;\n+\n+\t/* Process haves */\n+\tfor (i = 0; i < haves->nr; i++) {\n+\t\tconst struct object_id *oid = &haves->oid[i];\n+\t\tstruct object *o;\n+\t\tint we_knew_they_have = 0;\n+\n+\t\tif (!has_object_file(oid))\n+\t\t\tcontinue;\n+\n+\t\toid_array_append(common, oid);\n+\n+\t\to = parse_object(oid);\n+\t\tif (!o)\n+\t\t\tdie(\"oops (%s)\", oid_to_hex(oid));\n+\t\tif (o->type == OBJ_COMMIT) {\n+\t\t\tstruct commit_list *parents;\n+\t\t\tstruct commit *commit = (struct commit *)o;\n+\t\t\tif (o->flags & THEY_HAVE)\n+\t\t\t\twe_knew_they_have = 1;\n+\t\t\telse\n+\t\t\t\to->flags |= THEY_HAVE;\n+\t\t\tif (!oldest_have || (commit->date < oldest_have))\n+\t\t\t\toldest_have = commit->date;\n+\t\t\tfor (parents = commit->parents;\n+\t\t\t parents;\n+\t\t\t parents = parents->next)\n+\t\t\t\tparents->item->object.flags |= THEY_HAVE;\n+\t\t}\n+\t\tif (!we_knew_they_have)\n+\t\t\tadd_object_array(o, NULL, &have_obj);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int send_acks(struct oid_array *acks, struct strbuf *response)\n+{\n+\tint i;\n+\n+\tpacket_buf_write(response, \"acknowledgments\\n\");\n+\n+\t/* Send Acks */\n+\tif (!acks->nr)\n+\t\tpacket_buf_write(response, \"NAK\\n\");\n+\n+\tfor (i = 0; i < acks->nr; i++) {\n+\t\tpacket_buf_write(response, \"ACK %s\\n\",\n+\t\t\t\t oid_to_hex(&acks->oid[i]));\n+\t}\n+\n+\tif (ok_to_give_up()) {\n+\t\t/* Send Ready */\n+\t\tpacket_buf_write(response, \"ready\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_haves_and_send_acks(struct upload_pack_data *data)\n+{\n+\tstruct oid_array common = OID_ARRAY_INIT;\n+\tstruct strbuf response = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tprocess_haves(&data->haves, &common);\n+\tif (data->done) {\n+\t\tret = 1;\n+\t} else if (send_acks(&common, &response)) {\n+\t\tpacket_buf_delim(&response);\n+\t\tret = 1;\n+\t} else {\n+\t\t/* Add Flush */\n+\t\tpacket_buf_flush(&response);\n+\t\tret = 0;\n+\t}\n+\n+\t/* Send response */\n+\twrite_or_die(1, response.buf, response.len);\n+\tstrbuf_release(&response);\n+\n+\toid_array_clear(&data->haves);\n+\toid_array_clear(&common);\n+\treturn ret;\n+}\n+\n+enum fetch_state {\n+\tFETCH_PROCESS_ARGS = 0,\n+\tFETCH_READ_HAVES,\n+\tFETCH_SEND_ACKS,\n+\tFETCH_SEND_PACK,\n+\tFETCH_DONE,\n+};\n+\n+int upload_pack_v2(struct repository *r, struct argv_array *keys,\n+\t\t struct argv_array *args)\n+{\n+\tenum fetch_state state = FETCH_PROCESS_ARGS;\n+\tstruct upload_pack_data data = UPLOAD_PACK_DATA_INIT;\n+\tconst char *out;\n+\tuse_sideband = LARGE_PACKET_MAX;\n+\n+\t/* Check if cmd is being run as a stateless-rpc */\n+\tif (has_capability(keys, \"stateless-rpc\", &out))\n+\t\tif (!strcmp(out, \"true\"))\n+\t\t\tdata.stateless_rpc = 1;\n+\n+\twhile (state != FETCH_DONE) {\n+\t\tswitch (state) {\n+\t\tcase FETCH_PROCESS_ARGS:\n+\t\t\tprocess_args(args, &data);\n+\n+\t\t\tif (!want_obj.nr) {\n+\t\t\t\t/*\n+\t\t\t\t * Request didn't contain any 'want' lines,\n+\t\t\t\t * guess they didn't want anything.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\t} else if (data.haves.nr) {\n+\t\t\t\t/*\n+\t\t\t\t * Request had 'have' lines, so lets ACK them.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_SEND_ACKS;\n+\t\t\t} else {\n+\t\t\t\t/*\n+\t\t\t\t * Request had 'want's but no 'have's so we can\n+\t\t\t\t * immedietly go to construct and send a pack.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_SEND_PACK;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase FETCH_READ_HAVES:\n+\t\t\tread_haves(&data);\n+\t\t\tstate = FETCH_SEND_ACKS;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_ACKS:\n+\t\t\tif (process_haves_and_send_acks(&data))\n+\t\t\t\tstate = FETCH_SEND_PACK;\n+\t\t\telse if (data.stateless_rpc)\n+\t\t\t\t/*\n+\t\t\t\t * Request was made via stateless-rpc and a\n+\t\t\t\t * packfile isn't ready to be created and sent.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_READ_HAVES;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_PACK:\n+\t\t\tpacket_write_fmt(1, \"packfile\\n\");\n+\t\t\tcreate_pack_file();\n+\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_DONE:\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n+\tupload_pack_data_clear(&data);\n+\treturn 0;\n+}\ndiff --git a/upload-pack.h b/upload-pack.h\nindex a71e4dc7e..6b7890238 100644\n--- a/upload-pack.h\n+++ b/upload-pack.h\n@@ -10,4 +10,9 @@ struct upload_pack_options {\n \n void upload_pack(struct upload_pack_options *options);\n \n+struct repository;\n+struct argv_array;\n+extern int upload_pack_v2(struct repository *r, struct argv_array *keys,\n+\t\t\t struct argv_array *args);\n+\n #endif /* UPLOAD_PACK_H */\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337446","messageId":"20180125235838.138135-22-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 21/27] fetch-pack: perform a fetch using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:32Z","receivedAt":"2018-01-26T00:00:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"When communicating with a v2 server, perform a fetch by requesting the\n'fetch' command.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 2 +-\n fetch-pack.c | 277 ++++++++++++++++++++++++++++++++++++++++++++++++-\n fetch-pack.h | 4 +-\n t/t5702-protocol-v2.sh | 40 +++++++\n transport.c | 8 +-\n 5 files changed, 324 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex f492e8abd..867dd3cc7 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -213,7 +213,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n-\t\t\t &shallow, pack_lockfile_ptr);\n+\t\t\t &shallow, pack_lockfile_ptr, protocol_v0);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 9f6b07ad9..17927ae99 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1008,6 +1008,272 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \treturn ref;\n }\n \n+static void add_wants(const struct ref *wants, struct strbuf *req_buf)\n+{\n+\tfor ( ; wants ; wants = wants->next) {\n+\t\tconst struct object_id *remote = &wants->old_oid;\n+\t\tconst char *remote_hex;\n+\t\tstruct object *o;\n+\n+\t\t/*\n+\t\t * If that object is complete (i.e. it is an ancestor of a\n+\t\t * local ref), we tell them we have it but do not have to\n+\t\t * tell them about its ancestors, which they already know\n+\t\t * about.\n+\t\t *\n+\t\t * We use lookup_object here because we are only\n+\t\t * interested in the case we *know* the object is\n+\t\t * reachable and we have already scanned it.\n+\t\t */\n+\t\tif (((o = lookup_object(remote->hash)) != NULL) &&\n+\t\t (o->flags & COMPLETE)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tremote_hex = oid_to_hex(remote);\n+\t\tpacket_buf_write(req_buf, \"want %s\\n\", remote_hex);\n+\t}\n+}\n+\n+static int add_haves(struct strbuf *req_buf, int *in_vain)\n+{\n+\tint ret = 0;\n+\tint haves_added = 0;\n+\tconst struct object_id *oid;\n+\n+\twhile ((oid = get_rev())) {\n+\t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n+\t\tif (++haves_added >= INITIAL_FLUSH)\n+\t\t\tbreak;\n+\t};\n+\n+\t*in_vain += haves_added;\n+\tif (!haves_added || *in_vain >= MAX_IN_VAIN) {\n+\t\t/* Send Done */\n+\t\tpacket_buf_write(req_buf, \"done\\n\");\n+\t\tret = 1;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static int send_haves(int fd_out, int *in_vain)\n+{\n+\tint ret = 0;\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\n+\tret = add_haves(&req_buf, in_vain);\n+\n+\t/* Send request */\n+\tpacket_buf_flush(&req_buf);\n+\twrite_or_die(fd_out, req_buf.buf, req_buf.len);\n+\n+\tstrbuf_release(&req_buf);\n+\treturn ret;\n+}\n+\n+static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n+\t\t\t const struct ref *wants, struct oidset *common,\n+\t\t\t int *in_vain)\n+{\n+\tint ret = 0;\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\n+\tpacket_buf_write(&req_buf, \"command=fetch\");\n+\tpacket_buf_write(&req_buf, \"agent=%s\", git_user_agent_sanitized());\n+\tif (args->stateless_rpc)\n+\t\tpacket_buf_write(&req_buf, \"stateless-rpc=true\");\n+\n+\tpacket_buf_delim(&req_buf);\n+\tif (args->use_thin_pack)\n+\t\tpacket_buf_write(&req_buf, \"thin-pack\");\n+\tif (args->no_progress)\n+\t\tpacket_buf_write(&req_buf, \"no-progress\");\n+\tif (args->include_tag)\n+\t\tpacket_buf_write(&req_buf, \"include-tag\");\n+\tif (prefer_ofs_delta)\n+\t\tpacket_buf_write(&req_buf, \"ofs-delta\");\n+\n+\t/* add wants */\n+\tadd_wants(wants, &req_buf);\n+\n+\t/*\n+\t * If we are running stateless-rpc we need to add all the common\n+\t * commits we've found in previous rounds\n+\t */\n+\tif (args->stateless_rpc) {\n+\t\tstruct oidset_iter iter;\n+\t\tconst struct object_id *oid;\n+\t\toidset_iter_init(common, &iter);\n+\n+\t\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n+\t\t}\n+\t}\n+\n+\t/* Add initial haves */\n+\tret = add_haves(&req_buf, in_vain);\n+\n+\t/* Send request */\n+\tpacket_buf_flush(&req_buf);\n+\twrite_or_die(fd_out, req_buf.buf, req_buf.len);\n+\n+\tstrbuf_release(&req_buf);\n+\treturn ret;\n+}\n+\n+/*\n+ * Processes a section header in a server's response and checks if it matches\n+ * `section`. If the value of `peek` is 1, the header line will be peeked (and\n+ * not consumed); if 0, the line will be consumed and the function will die if\n+ * the section header doesn't match what was expected.\n+ */\n+static int process_section_header(struct packet_reader *reader,\n+\t\t\t\t const char *section, int peek)\n+{\n+\tint ret;\n+\n+\tif (packet_reader_peek(reader) != PACKET_READ_NORMAL)\n+\t\tdie(\"error reading packet\");\n+\n+\tret = !strcmp(reader->line, section);\n+\n+\tif (!peek) {\n+\t\tif (!ret)\n+\t\t\tdie(\"expected '%s', received '%s'\",\n+\t\t\t section, reader->line);\n+\t\tpacket_reader_read(reader);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static int process_acks(struct packet_reader *reader, struct oidset *common)\n+{\n+\t/* received */\n+\tint received_ready = 0;\n+\tint received_ack = 0;\n+\n+\tprocess_section_header(reader, \"acknowledgments\", 0);\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *arg;\n+\n+\t\tif (!strcmp(reader->line, \"NAK\"))\n+\t\t\tcontinue;\n+\n+\t\tif (skip_prefix(reader->line, \"ACK \", &arg)) {\n+\t\t\tstruct object_id oid;\n+\t\t\tif (!get_oid_hex(arg, &oid)) {\n+\t\t\t\tstruct commit *commit;\n+\t\t\t\toidset_insert(common, &oid);\n+\t\t\t\tcommit = lookup_commit(&oid);\n+\t\t\t\tmark_common(commit, 0, 1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!strcmp(reader->line, \"ready\")) {\n+\t\t\tclear_prio_queue(&rev_list);\n+\t\t\treceived_ready = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tdie(_(\"git fetch-pack: expected ACK/NAK, got '%s'\"), reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH &&\n+\t reader->status != PACKET_READ_DELIM)\n+\t\tdie(\"Error during processing acks: %d\", reader->status);\n+\n+\t/* return 0 if no common, 1 if there are common, or 2 if ready */\n+\treturn received_ready ? 2 : (received_ack ? 1 : 0);\n+}\n+\n+enum fetch_state {\n+\tFETCH_CHECK_LOCAL = 0,\n+\tFETCH_SEND_REQUEST,\n+\tFETCH_PROCESS_ACKS,\n+\tFETCH_SEND_HAVES,\n+\tFETCH_GET_PACK,\n+\tFETCH_DONE,\n+};\n+\n+static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n+\t\t\t\t int fd[2],\n+\t\t\t\t const struct ref *orig_ref,\n+\t\t\t\t struct ref **sought, int nr_sought,\n+\t\t\t\t char **pack_lockfile)\n+{\n+\tstruct ref *ref = copy_ref_list(orig_ref);\n+\tenum fetch_state state = FETCH_CHECK_LOCAL;\n+\tstruct oidset common = OIDSET_INIT;\n+\tstruct packet_reader reader;\n+\tint in_vain = 0;\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (state != FETCH_DONE) {\n+\t\tswitch (state) {\n+\t\tcase FETCH_CHECK_LOCAL:\n+\t\t\tsort_ref_list(&ref, ref_compare_name);\n+\t\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n+\n+\t\t\t/* v2 supports these by default */\n+\t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n+\t\t\tuse_sideband = 2;\n+\n+\t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n+\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_SEND_REQUEST;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_REQUEST:\n+\t\t\tif (send_fetch_request(fd[1], args, ref, &common, &in_vain))\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_PROCESS_ACKS;\n+\t\t\tbreak;\n+\t\tcase FETCH_PROCESS_ACKS:\n+\t\t\t/* Process ACKs/NAKs */\n+\t\t\tswitch (process_acks(&reader, &common)) {\n+\t\t\tcase 2:\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\t\tbreak;\n+\t\t\tcase 1:\n+\t\t\t\tin_vain = 0;\n+\t\t\t\t/* fallthrough */\n+\t\t\tdefault:\n+\t\t\t\tif (args->stateless_rpc)\n+\t\t\t\t\tstate = FETCH_SEND_REQUEST;\n+\t\t\t\telse\n+\t\t\t\t\tstate = FETCH_SEND_HAVES;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_HAVES:\n+\t\t\tif (send_haves(fd[1], &in_vain))\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_PROCESS_ACKS;\n+\t\t\tbreak;\n+\t\tcase FETCH_GET_PACK:\n+\t\t\t/* get the pack */\n+\t\t\tprocess_section_header(&reader, \"packfile\", 0);\n+\t\t\tif (get_pack(args, fd, pack_lockfile))\n+\t\t\t\tdie(_(\"git fetch-pack: fetch failed.\"));\n+\n+\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_DONE:\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n+\toidset_clear(&common);\n+\treturn ref;\n+}\n+\n static void fetch_pack_config(void)\n {\n \tgit_config_get_int(\"fetch.unpacklimit\", &fetch_unpack_limit);\n@@ -1153,7 +1419,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const char *dest,\n \t\t struct ref **sought, int nr_sought,\n \t\t struct oid_array *shallow,\n-\t\t char **pack_lockfile)\n+\t\t char **pack_lockfile,\n+\t\t enum protocol_version version)\n {\n \tstruct ref *ref_cpy;\n \tstruct shallow_info si;\n@@ -1167,8 +1434,12 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tdie(_(\"no matching remote head\"));\n \t}\n \tprepare_shallow_info(&si, shallow);\n-\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n-\t\t\t\t&si, pack_lockfile);\n+\tif (version == protocol_v2)\n+\t\tref_cpy = do_fetch_pack_v2(args, fd, ref, sought, nr_sought,\n+\t\t\t\t\t pack_lockfile);\n+\telse\n+\t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n+\t\t\t\t\t&si, pack_lockfile);\n \treprepare_packed_git();\n \tupdate_shallow(args, sought, nr_sought, &si);\n \tclear_shallow_info(&si);\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex b6aeb43a8..7afca7305 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -3,6 +3,7 @@\n \n #include \"string-list.h\"\n #include \"run-command.h\"\n+#include \"protocol.h\"\n \n struct oid_array;\n \n@@ -43,7 +44,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t struct ref **sought,\n \t\t int nr_sought,\n \t\t struct oid_array *shallow,\n-\t\t char **pack_lockfile);\n+\t\t char **pack_lockfile,\n+\t\t enum protocol_version version);\n \n /*\n * Print an appropriate error message for each sought ref that wasn't\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 7d8aeb766..3e411e178 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -33,4 +33,44 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol\n \t! grep \"refs/tags/\" log\n '\n \n+test_expect_success 'clone with file:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tclone \"file://$(pwd)/file_parent\" file_child 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 2\" log\n+'\n+\n+test_expect_success 'fetch with file:// using protocol v2' '\n+\ttest_commit -C file_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n+\t\tfetch origin 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 2\" log\n+'\n+\n+test_expect_success 'ref advertisment is filtered during fetch using protocol v2' '\n+\ttest_commit -C file_parent three &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n+\t\tfetch origin master 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgrep \"ref-pattern master\" log &&\n+\t! grep \"refs/tags/\" log\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex 6ea3905e3..4fdbd9adc 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -256,14 +256,18 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\trefs = fetch_pack(&args, data->fd, data->conn,\n+\t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n+\t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n+\t\t\t\t &transport->pack_lockfile, data->version);\n+\t\tpacket_flush(data->fd[1]);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n \t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n-\t\t\t\t &transport->pack_lockfile);\n+\t\t\t\t &transport->pack_lockfile, data->version);\n \t\tbreak;\n \tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337447","messageId":"20180125235838.138135-12-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 11/27] test-pkt-line: introduce a packet-line test helper","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:22Z","receivedAt":"2018-01-26T00:00:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce a packet-line test helper which can either pack or unpack an\ninput stream into packet-lines and writes out the result to stdout.\n\nSigned-off-by: Brandon Williams \n---\n Makefile | 1 +\n t/helper/test-pkt-line.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 63 insertions(+)\n create mode 100644 t/helper/test-pkt-line.c\n\ndiff --git a/Makefile b/Makefile\nindex b7ccc05fa..3b849c060 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -669,6 +669,7 @@ TEST_PROGRAMS_NEED_X += test-mktemp\n TEST_PROGRAMS_NEED_X += test-online-cpus\n TEST_PROGRAMS_NEED_X += test-parse-options\n TEST_PROGRAMS_NEED_X += test-path-utils\n+TEST_PROGRAMS_NEED_X += test-pkt-line\n TEST_PROGRAMS_NEED_X += test-prio-queue\n TEST_PROGRAMS_NEED_X += test-read-cache\n TEST_PROGRAMS_NEED_X += test-write-cache\ndiff --git a/t/helper/test-pkt-line.c b/t/helper/test-pkt-line.c\nnew file mode 100644\nindex 000000000..5df32b4cb\n--- /dev/null\n+++ b/t/helper/test-pkt-line.c\n@@ -0,0 +1,62 @@\n+#include \"pkt-line.h\"\n+\n+static void pack_line(const char *line)\n+{\n+\tif (!strcmp(line, \"0000\") || !strcmp(line, \"0000\\n\"))\n+\t\tpacket_flush(1);\n+\telse if (!strcmp(line, \"0001\") || !strcmp(line, \"0001\\n\"))\n+\t\tpacket_delim(1);\n+\telse\n+\t\tpacket_write_fmt(1, \"%s\", line);\n+}\n+\n+static void pack(int argc, const char **argv)\n+{\n+\tif (argc) { /* read from argv */\n+\t\tint i;\n+\t\tfor (i = 0; i < argc; i++)\n+\t\t\tpack_line(argv[i]);\n+\t} else { /* read from stdin */\n+\t\tchar line[LARGE_PACKET_MAX];\n+\t\twhile (fgets(line, sizeof(line), stdin)) {\n+\t\t\tpack_line(line);\n+\t\t}\n+\t}\n+}\n+\n+static void unpack(void)\n+{\n+\tstruct packet_reader reader;\n+\tpacket_reader_init(&reader, 0, NULL, 0,\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (packet_reader_read(&reader) != PACKET_READ_EOF) {\n+\t\tswitch (reader.status) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tprintf(\"%s\\n\", reader.line);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tprintf(\"0000\\n\");\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tprintf(\"0001\\n\");\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tif (argc < 2)\n+\t\tdie(\"too few arguments\");\n+\n+\tif (!strcmp(argv[1], \"pack\"))\n+\t\tpack(argc - 2, argv + 2);\n+\telse if (!strcmp(argv[1], \"unpack\"))\n+\t\tunpack();\n+\n+\treturn 0;\n+}\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337448","messageId":"20180125235838.138135-6-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v2 05/27] upload-pack: factor out processing lines","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-25T23:58:16Z","receivedAt":"2018-01-26T00:00:14Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Factor out the logic for processing shallow, deepen, deepen_since, and\ndeepen_not lines into their own functions to simplify the\n'receive_needs()' function in addition to making it easier to reuse some\nof this logic when implementing protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 74 insertions(+), 39 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 2ad73a98b..42d83d5b1 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,\n \tpacket_flush(1);\n }\n \n+static int process_shallow(const char *line, struct object_array *shallows)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"shallow \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tstruct object *object;\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"invalid shallow line: %s\", line);\n+\t\tobject = parse_object(&oid);\n+\t\tif (!object)\n+\t\t\treturn 1;\n+\t\tif (object->type != OBJ_COMMIT)\n+\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n+\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n+\t\t\tobject->flags |= CLIENT_SHALLOW;\n+\t\t\tadd_object_array(object, NULL, shallows);\n+\t\t}\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_deepen(const char *line, int *depth)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\t*depth = (int) strtol(arg, &end, 0);\n+\t\tif (!end || *end || *depth <= 0)\n+\t\t\tdie(\"Invalid deepen: %s\", line);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen-since \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\t*deepen_since = parse_timestamp(arg, &end, 0);\n+\t\tif (!end || *end || !deepen_since ||\n+\t\t /* revisions.c's max_age -1 is special */\n+\t\t *deepen_since == -1)\n+\t\t\tdie(\"Invalid deepen-since: %s\", line);\n+\t\t*deepen_rev_list = 1;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen-not \", &arg)) {\n+\t\tchar *ref = NULL;\n+\t\tstruct object_id oid;\n+\t\tif (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n+\t\t\tdie(\"git upload-pack: ambiguous deepen-not: %s\", line);\n+\t\tstring_list_append(deepen_not, ref);\n+\t\tfree(ref);\n+\t\t*deepen_rev_list = 1;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static void receive_needs(void)\n {\n \tstruct object_array shallows = OBJECT_ARRAY_INIT;\n@@ -745,49 +814,15 @@ static void receive_needs(void)\n \t\tif (!line)\n \t\t\tbreak;\n \n-\t\tif (skip_prefix(line, \"shallow \", &arg)) {\n-\t\t\tstruct object_id oid;\n-\t\t\tstruct object *object;\n-\t\t\tif (get_oid_hex(arg, &oid))\n-\t\t\t\tdie(\"invalid shallow line: %s\", line);\n-\t\t\tobject = parse_object(&oid);\n-\t\t\tif (!object)\n-\t\t\t\tcontinue;\n-\t\t\tif (object->type != OBJ_COMMIT)\n-\t\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n-\t\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n-\t\t\t\tobject->flags |= CLIENT_SHALLOW;\n-\t\t\t\tadd_object_array(object, NULL, &shallows);\n-\t\t\t}\n+\t\tif (process_shallow(line, &shallows))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen \", &arg)) {\n-\t\t\tchar *end = NULL;\n-\t\t\tdepth = strtol(arg, &end, 0);\n-\t\t\tif (!end || *end || depth <= 0)\n-\t\t\t\tdie(\"Invalid deepen: %s\", line);\n+\t\tif (process_deepen(line, &depth))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen-since \", &arg)) {\n-\t\t\tchar *end = NULL;\n-\t\t\tdeepen_since = parse_timestamp(arg, &end, 0);\n-\t\t\tif (!end || *end || !deepen_since ||\n-\t\t\t /* revisions.c's max_age -1 is special */\n-\t\t\t deepen_since == -1)\n-\t\t\t\tdie(\"Invalid deepen-since: %s\", line);\n-\t\t\tdeepen_rev_list = 1;\n+\t\tif (process_deepen_since(line, &deepen_since, &deepen_rev_list))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen-not \", &arg)) {\n-\t\t\tchar *ref = NULL;\n-\t\t\tstruct object_id oid;\n-\t\t\tif (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n-\t\t\t\tdie(\"git upload-pack: ambiguous deepen-not: %s\", line);\n-\t\t\tstring_list_append(&deepen_not, ref);\n-\t\t\tfree(ref);\n-\t\t\tdeepen_rev_list = 1;\n+\t\tif (process_deepen_not(line, &deepen_not, &deepen_rev_list))\n \t\t\tcontinue;\n-\t\t}\n+\n \t\tif (!skip_prefix(line, \"want \", &arg) ||\n \t\t get_oid_hex(arg, &oid_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"337459","messageId":"CACsJy8BHh+Kv=om+rc6=TM60E1j9=wq4rRBvKB8HLfNTMa51Rg@mail.gmail.com","threadId":"47524","inReplyTo":"20180125235838.138135-13-bmwill@google.com","subject":"Re: [PATCH v2 12/27] serve: introduce git-serve","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-26T10:39:51Z","receivedAt":"2018-01-26T10:40:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jan 26, 2018 at 6:58 AM, Brandon Williams wrote:\n> + Detailed Design\n> +=================\n> +\n> +A client can request to speak protocol v2 by sending `version=2` in the\n> +side-channel `GIT_PROTOCOL` in the initial request to the server.\n> +\n> +In protocol v2 communication is command oriented. When first contacting a\n> +server a list of capabilities will advertised. Some of these capabilities\n\ns/will advertised/will be advertised/\n\n> + Capability Advertisement\n> +--------------------------\n> +\n> +A server which decides to communicate (based on a request from a client)\n> +using protocol version 2, notifies the client by sending a version string\n> +in its initial response followed by an advertisement of its capabilities.\n> +Each capability is a key with an optional value. Clients must ignore all\n> +unknown keys.\n\nWith have a convention in $GIT_DIR/index file format that's probably a\ngood thing to follow here: lowercase keys are optional, such unknown\nkeys can (and must) be ignored. Uppercase keys are mandatory. If a\nclient can't understand one of those keys, abort. This gives the\nserver a way to \"select\" clients and introduce incompatible changes if\nwe ever have to.\n\n> Semantics of unknown values are left to the definition of\n> +each key. Some capabilities will describe commands which can be requested\n> +to be executed by the client.\n> +\n> + capability-advertisement = protocol-version\n> + capability-list\n> + flush-pkt\n> +\n> + protocol-version = PKT-LINE(\"version 2\" LF)\n> + capability-list = *capability\n> + capability = PKT-LINE(key[=value] LF)\n> +\n> + key = 1*CHAR\n> + value = 1*CHAR\n> + CHAR = 1*(ALPHA / DIGIT / \"-\" / \"_\")\n\nIs this a bit too restricted for \"value\"? Something like \".\" (e.g.\nversion) or \"@\" (I wonder if anybody will add an capability that\ncontains an email address). Unless there's a good reason to limit it,\nshould we just go full ascii (without control codes)?\n\n> +A client then responds to select the command it wants with any particular\n> +capabilities or arguments. There is then an optional section where the\n> +client can provide any command specific parameters or queries.\n> +\n> + command-request = command\n> + capability-list\n> + (command-args)\n> + flush-pkt\n> + command = PKT-LINE(\"command=\" key LF)\n> + command-args = delim-pkt\n> + *arg\n> + arg = 1*CHAR\n> +\n> +The server will then check to ensure that the client's request is\n> +comprised of a valid command as well as valid capabilities which were\n> +advertised. If the request is valid the server will then execute the\n> +command.\n\nWhat happens when the request is not valid? Or..\n\n> +When a command has finished\n\nHow does the client know a command has finished? Is it up to each\ncommand design?\n\nMore or less related it bugs me that I have a translated git client,\nbut I still receive remote error messages in English. It's a hard\nproblem, but I'm hoping that we won't need to change the core protocol\nto support that someday. Although we could make rule now that side\nchannel message could be sent in \"printf\"-like form, where the client\ncan translate the format string and substitutes placeholders with real\nvalues afterward...\n\n> a client can either request that another\n> +command be executed or can terminate the connection by sending an empty\n> +request consisting of just a flush-pkt.\n> +\n> + Capabilities\n> +~~~~~~~~~~~~~~\n> +\n> +There are two different types of capabilities: normal capabilities,\n> +which can be used to to convey information or alter the behavior of a\n> +request, and command capabilities, which are the core actions that a\n> +client wants to perform (fetch, push, etc).\n> +\n> + agent\n> +-------\n> +\n> +The server can advertise the `agent` capability with a value `X` (in the\n> +form `agent=X`) to notify the client that the server is running version\n> +`X`. The client may optionally send its own agent string by including\n> +the `agent` capability with a value `Y` (in the form `agent=Y`) in its\n> +request to the server (but it MUST NOT do so if the server did not\n> +advertise the agent capability). The `X` and `Y` strings may contain any\n> +printable ASCII characters except space (i.e., the byte range 32 < x <\n> +127), and are typically of the form \"package/version\" (e.g.,\n> +\"git/1.8.3.1\"). The agent strings are purely informative for statistics\n> +and debugging purposes, and MUST NOT be used to programmatically assume\n> +the presence or absence of particular features.\n> +\n> + stateless-rpc\n> +---------------\n> +\n> +If advertised, the `stateless-rpc` capability indicates that the server\n> +supports running commands in a stateless-rpc mode, which means that a\n> +command lasts for only a single request-response round.\n> +\n> +Normally a command can last for as many rounds as are required to\n> +complete it (multiple for negotiation during fetch or no additional\n> +trips in the case of ls-refs). If the client sends the `stateless-rpc`\n> +capability with a value of `true` (in the form `stateless-rpc=true`)\n> +then the invoked command must only last a single round.\n\nSpeaking of stateless-rpc, I remember last time this topic was brought\nup, there was some discussion to kind of optimize it for http as well,\nto fit the \"client sends request, server responds data\" model and\navoid too many round trips (ideally everything happens in one round\ntrip). Does it evolve to anything real? All the cool stuff happened\nwhile I was away, sorry if this was discussed and settled.\n-- \nDuy\n"},{"id":"337544","messageId":"CAGZ79kYbhR-y7WkhFgQ-YBkWaNXWuadDAGXaAzWDyJBzDSqc2w@mail.gmail.com","threadId":"47524","inReplyTo":"20180125235838.138135-6-bmwill@google.com","subject":"Re: [PATCH v2 05/27] upload-pack: factor out processing lines","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-26T20:12:09Z","receivedAt":"2018-01-26T20:12:16Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams wrote:\n> Factor out the logic for processing shallow, deepen, deepen_since, and\n> deepen_not lines into their own functions to simplify the\n> 'receive_needs()' function in addition to making it easier to reuse some\n> of this logic when implementing protocol_v2.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------\n> 1 file changed, 74 insertions(+), 39 deletions(-)\n>\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 2ad73a98b..42d83d5b1 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,\n> packet_flush(1);\n> }\n>\n> +static int process_shallow(const char *line, struct object_array *shallows)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"shallow \", &arg)) {\n\nstylistic nit:\n\n You could invert the condition in each of the process_* functions\n to just have\n\n if (!skip_prefix...))\n return 0\n\n /* less indented code goes here */\n\n return 1;\n\n That way we have less indentation as well as easier code.\n (The reader doesn't need to keep in mind what the else\n part is about; it is a rather local decision to bail out instead\n of having the return at the end of the function.)\n\n> + struct object_id oid;\n> + struct object *object;\n> + if (get_oid_hex(arg, &oid))\n> + die(\"invalid shallow line: %s\", line);\n> + object = parse_object(&oid);\n> + if (!object)\n> + return 1;\n> + if (object->type != OBJ_COMMIT)\n> + die(\"invalid shallow object %s\", oid_to_hex(&oid));\n> + if (!(object->flags & CLIENT_SHALLOW)) {\n> + object->flags |= CLIENT_SHALLOW;\n> + add_object_array(object, NULL, shallows);\n> + }\n> + return 1;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +static int process_deepen(const char *line, int *depth)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"deepen \", &arg)) {\n> + char *end = NULL;\n> + *depth = (int) strtol(arg, &end, 0);\n> + if (!end || *end || *depth <= 0)\n> + die(\"Invalid deepen: %s\", line);\n> + return 1;\n> + }\n> +\n> + return 0;\n> +}\n> +\n> +static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"deepen-since \", &arg)) {\n> + char *end = NULL;\n> + *deepen_since = parse_timestamp(arg, &end, 0);\n> + if (!end || *end || !deepen_since ||\n> + /* revisions.c's max_age -1 is special */\n> + *deepen_since == -1)\n> + die(\"Invalid deepen-since: %s\", line);\n> + *deepen_rev_list = 1;\n> + return 1;\n> + }\n> + return 0;\n> +}\n> +\n> +static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)\n> +{\n> + const char *arg;\n> + if (skip_prefix(line, \"deepen-not \", &arg)) {\n> + char *ref = NULL;\n> + struct object_id oid;\n> + if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n> + die(\"git upload-pack: ambiguous deepen-not: %s\", line);\n> + string_list_append(deepen_not, ref);\n> + free(ref);\n> + *deepen_rev_list = 1;\n> + return 1;\n> + }\n> + return 0;\n> +}\n> +\n> static void receive_needs(void)\n> {\n> struct object_array shallows = OBJECT_ARRAY_INIT;\n> @@ -745,49 +814,15 @@ static void receive_needs(void)\n> if (!line)\n> break;\n>\n> - if (skip_prefix(line, \"shallow \", &arg)) {\n> - struct object_id oid;\n> - struct object *object;\n> - if (get_oid_hex(arg, &oid))\n> - die(\"invalid shallow line: %s\", line);\n> - object = parse_object(&oid);\n> - if (!object)\n> - continue;\n> - if (object->type != OBJ_COMMIT)\n> - die(\"invalid shallow object %s\", oid_to_hex(&oid));\n> - if (!(object->flags & CLIENT_SHALLOW)) {\n> - object->flags |= CLIENT_SHALLOW;\n> - add_object_array(object, NULL, &shallows);\n> - }\n> + if (process_shallow(line, &shallows))\n> continue;\n> - }\n> - if (skip_prefix(line, \"deepen \", &arg)) {\n> - char *end = NULL;\n> - depth = strtol(arg, &end, 0);\n> - if (!end || *end || depth <= 0)\n> - die(\"Invalid deepen: %s\", line);\n> + if (process_deepen(line, &depth))\n> continue;\n> - }\n> - if (skip_prefix(line, \"deepen-since \", &arg)) {\n> - char *end = NULL;\n> - deepen_since = parse_timestamp(arg, &end, 0);\n> - if (!end || *end || !deepen_since ||\n> - /* revisions.c's max_age -1 is special */\n> - deepen_since == -1)\n> - die(\"Invalid deepen-since: %s\", line);\n> - deepen_rev_list = 1;\n> + if (process_deepen_since(line, &deepen_since, &deepen_rev_list))\n> continue;\n> - }\n> - if (skip_prefix(line, \"deepen-not \", &arg)) {\n> - char *ref = NULL;\n> - struct object_id oid;\n> - if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n> - die(\"git upload-pack: ambiguous deepen-not: %s\", line);\n> - string_list_append(&deepen_not, ref);\n> - free(ref);\n> - deepen_rev_list = 1;\n> + if (process_deepen_not(line, &deepen_not, &deepen_rev_list))\n> continue;\n> - }\n> +\n> if (!skip_prefix(line, \"want \", &arg) ||\n> get_oid_hex(arg, &oid_buf))\n> die(\"git upload-pack: protocol error, \"\n> --\n> 2.16.0.rc1.238.g530d649a79-goog\n>\n"},{"id":"337552","messageId":"20180126213338.GB17576@google.com","threadId":"47524","inReplyTo":"CAGZ79kYbhR-y7WkhFgQ-YBkWaNXWuadDAGXaAzWDyJBzDSqc2w@mail.gmail.com","subject":"Re: [PATCH v2 05/27] upload-pack: factor out processing lines","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-26T21:33:38Z","receivedAt":"2018-01-26T21:33:46Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/26, Stefan Beller wrote:\n> On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams wrote:\n> > Factor out the logic for processing shallow, deepen, deepen_since, and\n> > deepen_not lines into their own functions to simplify the\n> > 'receive_needs()' function in addition to making it easier to reuse some\n> > of this logic when implementing protocol_v2.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------\n> > 1 file changed, 74 insertions(+), 39 deletions(-)\n> >\n> > diff --git a/upload-pack.c b/upload-pack.c\n> > index 2ad73a98b..42d83d5b1 100644\n> > --- a/upload-pack.c\n> > +++ b/upload-pack.c\n> > @@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,\n> > packet_flush(1);\n> > }\n> >\n> > +static int process_shallow(const char *line, struct object_array *shallows)\n> > +{\n> > + const char *arg;\n> > + if (skip_prefix(line, \"shallow \", &arg)) {\n> \n> stylistic nit:\n> \n> You could invert the condition in each of the process_* functions\n> to just have\n> \n> if (!skip_prefix...))\n> return 0\n> \n> /* less indented code goes here */\n> \n> return 1;\n> \n> That way we have less indentation as well as easier code.\n> (The reader doesn't need to keep in mind what the else\n> part is about; it is a rather local decision to bail out instead\n> of having the return at the end of the function.)\n\nI was trying to move the existing code into helper functions so\nrewriting them in transit may make it less reviewable?\n\n> \n> > + struct object_id oid;\n> > + struct object *object;\n> > + if (get_oid_hex(arg, &oid))\n> > + die(\"invalid shallow line: %s\", line);\n> > + object = parse_object(&oid);\n> > + if (!object)\n> > + return 1;\n> > + if (object->type != OBJ_COMMIT)\n> > + die(\"invalid shallow object %s\", oid_to_hex(&oid));\n> > + if (!(object->flags & CLIENT_SHALLOW)) {\n> > + object->flags |= CLIENT_SHALLOW;\n> > + add_object_array(object, NULL, shallows);\n> > + }\n> > + return 1;\n> > + }\n> > +\n> > + return 0;\n> > +}\n> > +\n> > +static int process_deepen(const char *line, int *depth)\n> > +{\n> > + const char *arg;\n> > + if (skip_prefix(line, \"deepen \", &arg)) {\n> > + char *end = NULL;\n> > + *depth = (int) strtol(arg, &end, 0);\n> > + if (!end || *end || *depth <= 0)\n> > + die(\"Invalid deepen: %s\", line);\n> > + return 1;\n> > + }\n> > +\n> > + return 0;\n> > +}\n> > +\n> > +static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)\n> > +{\n> > + const char *arg;\n> > + if (skip_prefix(line, \"deepen-since \", &arg)) {\n> > + char *end = NULL;\n> > + *deepen_since = parse_timestamp(arg, &end, 0);\n> > + if (!end || *end || !deepen_since ||\n> > + /* revisions.c's max_age -1 is special */\n> > + *deepen_since == -1)\n> > + die(\"Invalid deepen-since: %s\", line);\n> > + *deepen_rev_list = 1;\n> > + return 1;\n> > + }\n> > + return 0;\n> > +}\n> > +\n> > +static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)\n> > +{\n> > + const char *arg;\n> > + if (skip_prefix(line, \"deepen-not \", &arg)) {\n> > + char *ref = NULL;\n> > + struct object_id oid;\n> > + if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n> > + die(\"git upload-pack: ambiguous deepen-not: %s\", line);\n> > + string_list_append(deepen_not, ref);\n> > + free(ref);\n> > + *deepen_rev_list = 1;\n> > + return 1;\n> > + }\n> > + return 0;\n> > +}\n> > +\n> > static void receive_needs(void)\n> > {\n> > struct object_array shallows = OBJECT_ARRAY_INIT;\n> > @@ -745,49 +814,15 @@ static void receive_needs(void)\n> > if (!line)\n> > break;\n> >\n> > - if (skip_prefix(line, \"shallow \", &arg)) {\n> > - struct object_id oid;\n> > - struct object *object;\n> > - if (get_oid_hex(arg, &oid))\n> > - die(\"invalid shallow line: %s\", line);\n> > - object = parse_object(&oid);\n> > - if (!object)\n> > - continue;\n> > - if (object->type != OBJ_COMMIT)\n> > - die(\"invalid shallow object %s\", oid_to_hex(&oid));\n> > - if (!(object->flags & CLIENT_SHALLOW)) {\n> > - object->flags |= CLIENT_SHALLOW;\n> > - add_object_array(object, NULL, &shallows);\n> > - }\n> > + if (process_shallow(line, &shallows))\n> > continue;\n> > - }\n> > - if (skip_prefix(line, \"deepen \", &arg)) {\n> > - char *end = NULL;\n> > - depth = strtol(arg, &end, 0);\n> > - if (!end || *end || depth <= 0)\n> > - die(\"Invalid deepen: %s\", line);\n> > + if (process_deepen(line, &depth))\n> > continue;\n> > - }\n> > - if (skip_prefix(line, \"deepen-since \", &arg)) {\n> > - char *end = NULL;\n> > - deepen_since = parse_timestamp(arg, &end, 0);\n> > - if (!end || *end || !deepen_since ||\n> > - /* revisions.c's max_age -1 is special */\n> > - deepen_since == -1)\n> > - die(\"Invalid deepen-since: %s\", line);\n> > - deepen_rev_list = 1;\n> > + if (process_deepen_since(line, &deepen_since, &deepen_rev_list))\n> > continue;\n> > - }\n> > - if (skip_prefix(line, \"deepen-not \", &arg)) {\n> > - char *ref = NULL;\n> > - struct object_id oid;\n> > - if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n> > - die(\"git upload-pack: ambiguous deepen-not: %s\", line);\n> > - string_list_append(&deepen_not, ref);\n> > - free(ref);\n> > - deepen_rev_list = 1;\n> > + if (process_deepen_not(line, &deepen_not, &deepen_rev_list))\n> > continue;\n> > - }\n> > +\n> > if (!skip_prefix(line, \"want \", &arg) ||\n> > get_oid_hex(arg, &oid_buf))\n> > die(\"git upload-pack: protocol error, \"\n> > --\n> > 2.16.0.rc1.238.g530d649a79-goog\n> >\n\n-- \nBrandon Williams\n"},{"id":"337557","messageId":"CAGZ79kb14ciUoyHLAxmh0jGwwPYFVjGayrn9R2f+4+88A8=xsA@mail.gmail.com","threadId":"47524","inReplyTo":"20180125235838.138135-14-bmwill@google.com","subject":"Re: [PATCH v2 13/27] ls-refs: introduce ls-refs server command","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-26T22:20:12Z","receivedAt":"2018-01-26T22:20:20Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams wrote:\n\n> +ls-refs takes in the following parameters wrapped in packet-lines:\n> +\n> + symrefs\n> + In addition to the object pointed by it, show the underlying ref\n> + pointed by it when showing a symbolic ref.\n> + peel\n> + Show peeled tags.\n\nWould it make sense to default these two to on, and rather have\noptional no-symrefs and no-peel ?\n\nThat would save bandwidth in the default case, I would think.\n\n> + cat >expect <<-EOF &&\n> + $(git rev-parse HEAD) HEAD\n> + $(git rev-parse refs/heads/dev) refs/heads/dev\n> + $(git rev-parse refs/heads/master) refs/heads/master\n> + $(git rev-parse refs/heads/release) refs/heads/release\n> + $(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag\n> + $(git rev-parse refs/tags/one) refs/tags/one\n> + $(git rev-parse refs/tags/two) refs/tags/two\n\nInvoking rev-parse quite a few times? I think the test suite is a\ntrade off between readability (\"what we expect the test to do and test\")\nand speed (specifically on Windows forking is expensive);\n\nI tried to come up with a more concise way to create this expectation\nusing git-rev-parse, but did not find a good way to do so.\n\nHowever maybe\n\n git for-each-ref --format='%(*objectname) %(*refname)' >expect\n\nmight help in reproducing the expected message? The downside\nof this would be to have to closely guard which refs are there though.\nI guess the '--pattern' could help there as it may be the same pattern\nas the input to the ls-refs. This might be too abstract for a test though.\n\nI dunno.\n\nStefan\n"},{"id":"337956","messageId":"8df41c05-c1bf-b834-2815-0dfe6b1c7b09@gmail.com","threadId":"47524","inReplyTo":"20180126213338.GB17576@google.com","subject":"Re: [PATCH v2 05/27] upload-pack: factor out processing lines","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-31T14:08:27Z","receivedAt":"2018-01-31T14:08:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/26/2018 4:33 PM, Brandon Williams wrote:\n> On 01/26, Stefan Beller wrote:\n>> On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams wrote:\n>>> Factor out the logic for processing shallow, deepen, deepen_since, and\n>>> deepen_not lines into their own functions to simplify the\n>>> 'receive_needs()' function in addition to making it easier to reuse some\n>>> of this logic when implementing protocol_v2.\n>>>\n>>> Signed-off-by: Brandon Williams \n>>> ---\n>>> upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------\n>>> 1 file changed, 74 insertions(+), 39 deletions(-)\n>>>\n>>> diff --git a/upload-pack.c b/upload-pack.c\n>>> index 2ad73a98b..42d83d5b1 100644\n>>> --- a/upload-pack.c\n>>> +++ b/upload-pack.c\n>>> @@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,\n>>> packet_flush(1);\n>>> }\n>>>\n>>> +static int process_shallow(const char *line, struct object_array *shallows)\n>>> +{\n>>> + const char *arg;\n>>> + if (skip_prefix(line, \"shallow \", &arg)) {\n>> stylistic nit:\n>>\n>> You could invert the condition in each of the process_* functions\n>> to just have\n>>\n>> if (!skip_prefix...))\n>> return 0\n>>\n>> /* less indented code goes here */\n>>\n>> return 1;\n>>\n>> That way we have less indentation as well as easier code.\n>> (The reader doesn't need to keep in mind what the else\n>> part is about; it is a rather local decision to bail out instead\n>> of having the return at the end of the function.)\n> I was trying to move the existing code into helper functions so\n> rewriting them in transit may make it less reviewable?\n\nI think the way you kept to the existing code as much as possible is \ngood and easier to review. Perhaps a style pass after the patch lands is \ngood for #leftoverbits.\n\n>>\n>>> + struct object_id oid;\n>>> + struct object *object;\n>>> + if (get_oid_hex(arg, &oid))\n>>> + die(\"invalid shallow line: %s\", line);\n>>> + object = parse_object(&oid);\n>>> + if (!object)\n>>> + return 1;\n>>> + if (object->type != OBJ_COMMIT)\n>>> + die(\"invalid shallow object %s\", oid_to_hex(&oid));\n>>> + if (!(object->flags & CLIENT_SHALLOW)) {\n>>> + object->flags |= CLIENT_SHALLOW;\n>>> + add_object_array(object, NULL, shallows);\n>>> + }\n>>> + return 1;\n>>> + }\n>>> +\n>>> + return 0;\n>>> +}\n>>> +\n>>> +static int process_deepen(const char *line, int *depth)\n>>> +{\n>>> + const char *arg;\n>>> + if (skip_prefix(line, \"deepen \", &arg)) {\n>>> + char *end = NULL;\n>>> + *depth = (int) strtol(arg, &end, 0);\n\nnit: space between (int) and strtol?\n\n>>> + if (!end || *end || *depth <= 0)\n>>> + die(\"Invalid deepen: %s\", line);\n>>> + return 1;\n>>> + }\n>>> +\n>>> + return 0;\n>>> +}\n>>> +\n>>> +static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)\n>>> +{\n>>> + const char *arg;\n>>> + if (skip_prefix(line, \"deepen-since \", &arg)) {\n>>> + char *end = NULL;\n>>> + *deepen_since = parse_timestamp(arg, &end, 0);\n>>> + if (!end || *end || !deepen_since ||\n>>> + /* revisions.c's max_age -1 is special */\n>>> + *deepen_since == -1)\n>>> + die(\"Invalid deepen-since: %s\", line);\n>>> + *deepen_rev_list = 1;\n>>> + return 1;\n>>> + }\n>>> + return 0;\n>>> +}\n>>> +\n>>> +static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)\n>>> +{\n>>> + const char *arg;\n>>> + if (skip_prefix(line, \"deepen-not \", &arg)) {\n>>> + char *ref = NULL;\n>>> + struct object_id oid;\n>>> + if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n>>> + die(\"git upload-pack: ambiguous deepen-not: %s\", line);\n>>> + string_list_append(deepen_not, ref);\n>>> + free(ref);\n>>> + *deepen_rev_list = 1;\n>>> + return 1;\n>>> + }\n>>> + return 0;\n>>> +}\n>>> +\n>>> static void receive_needs(void)\n>>> {\n>>> struct object_array shallows = OBJECT_ARRAY_INIT;\n>>> @@ -745,49 +814,15 @@ static void receive_needs(void)\n>>> if (!line)\n>>> break;\n>>>\n>>> - if (skip_prefix(line, \"shallow \", &arg)) {\n>>> - struct object_id oid;\n>>> - struct object *object;\n>>> - if (get_oid_hex(arg, &oid))\n>>> - die(\"invalid shallow line: %s\", line);\n>>> - object = parse_object(&oid);\n>>> - if (!object)\n>>> - continue;\n>>> - if (object->type != OBJ_COMMIT)\n>>> - die(\"invalid shallow object %s\", oid_to_hex(&oid));\n>>> - if (!(object->flags & CLIENT_SHALLOW)) {\n>>> - object->flags |= CLIENT_SHALLOW;\n>>> - add_object_array(object, NULL, &shallows);\n>>> - }\n>>> + if (process_shallow(line, &shallows))\n>>> continue;\n>>> - }\n>>> - if (skip_prefix(line, \"deepen \", &arg)) {\n>>> - char *end = NULL;\n>>> - depth = strtol(arg, &end, 0);\n>>> - if (!end || *end || depth <= 0)\n>>> - die(\"Invalid deepen: %s\", line);\n>>> + if (process_deepen(line, &depth))\n>>> continue;\n>>> - }\n>>> - if (skip_prefix(line, \"deepen-since \", &arg)) {\n>>> - char *end = NULL;\n>>> - deepen_since = parse_timestamp(arg, &end, 0);\n>>> - if (!end || *end || !deepen_since ||\n>>> - /* revisions.c's max_age -1 is special */\n>>> - deepen_since == -1)\n>>> - die(\"Invalid deepen-since: %s\", line);\n>>> - deepen_rev_list = 1;\n>>> + if (process_deepen_since(line, &deepen_since, &deepen_rev_list))\n>>> continue;\n>>> - }\n>>> - if (skip_prefix(line, \"deepen-not \", &arg)) {\n>>> - char *ref = NULL;\n>>> - struct object_id oid;\n>>> - if (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n>>> - die(\"git upload-pack: ambiguous deepen-not: %s\", line);\n>>> - string_list_append(&deepen_not, ref);\n>>> - free(ref);\n>>> - deepen_rev_list = 1;\n>>> + if (process_deepen_not(line, &deepen_not, &deepen_rev_list))\n>>> continue;\n>>> - }\n>>> +\n>>> if (!skip_prefix(line, \"want \", &arg) ||\n>>> get_oid_hex(arg, &oid_buf))\n>>> die(\"git upload-pack: protocol error, \"\n>>> --\n>>> 2.16.0.rc1.238.g530d649a79-goog\n>>>\n\n"},{"id":"337957","messageId":"223c0249-24e3-d714-faa5-3c7166b92e99@gmail.com","threadId":"47524","inReplyTo":"20180125235838.138135-9-bmwill@google.com","subject":"Re: [PATCH v2 08/27] connect: discover protocol version outside of get_remote_heads","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-31T14:40:29Z","receivedAt":"2018-01-31T14:40:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/25/2018 6:58 PM, Brandon Williams wrote:\n> In order to prepare for the addition of protocol_v2 push the protocol\n> version discovery outside of 'get_remote_heads()'. This will allow for\n> keeping the logic for processing the reference advertisement for\n> protocol_v1 and protocol_v0 separate from the logic for protocol_v2.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> builtin/fetch-pack.c | 16 +++++++++++++++-\n> builtin/send-pack.c | 17 +++++++++++++++--\n> connect.c | 27 ++++++++++-----------------\n> connect.h | 3 +++\n> remote-curl.c | 20 ++++++++++++++++++--\n> remote.h | 5 +++--\n> transport.c | 24 +++++++++++++++++++-----\n> 7 files changed, 83 insertions(+), 29 deletions(-)\n>\n> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n> index 366b9d13f..85d4faf76 100644\n> --- a/builtin/fetch-pack.c\n> +++ b/builtin/fetch-pack.c\n> @@ -4,6 +4,7 @@\n> #include \"remote.h\"\n> #include \"connect.h\"\n> #include \"sha1-array.h\"\n> +#include \"protocol.h\"\n> \n> static const char fetch_pack_usage[] =\n> \"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] \"\n> @@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> \tstruct fetch_pack_args args;\n> \tstruct oid_array shallow = OID_ARRAY_INIT;\n> \tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n> +\tstruct packet_reader reader;\n> \n> \tpacket_trace_identity(\"fetch-pack\");\n> \n> @@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> \t\tif (!conn)\n> \t\t\treturn args.diag_url ? 0 : 1;\n> \t}\n> -\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n> +\n> +\tpacket_reader_init(&reader, fd[0], NULL, 0,\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> +\n> +\tswitch (discover_version(&reader)) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n\nIs this really a BUG in the client, or a bug/incompatibility in the server?\n\nPerhaps I'm misunderstanding, but it looks like discover_version() will \ndie() on an unknown version (the die() is in \nprotocol.c:determine_protocol_version_client()). So maybe that's why \nthis is a BUG()?\n\nIf there is something to change here, this BUG() appears three more times.\n\n> +\t}\n> \n> \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n> \t\t\t &shallow, pack_lockfile_ptr);\n> diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n> index fc4f0bb5f..83cb125a6 100644\n> --- a/builtin/send-pack.c\n> +++ b/builtin/send-pack.c\n> @@ -14,6 +14,7 @@\n> #include \"sha1-array.h\"\n> #include \"gpg-interface.h\"\n> #include \"gettext.h\"\n> +#include \"protocol.h\"\n> \n> static const char * const send_pack_usage[] = {\n> \tN_(\"git send-pack [--all | --mirror] [--dry-run] [--force] \"\n> @@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n> \tint progress = -1;\n> \tint from_stdin = 0;\n> \tstruct push_cas_option cas = {0};\n> +\tstruct packet_reader reader;\n> \n> \tstruct option options[] = {\n> \t\tOPT__VERBOSITY(&verbose),\n> @@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n> \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n> \t}\n> \n> -\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n> -\t\t\t &extra_have, &shallow);\n> +\tpacket_reader_init(&reader, fd[0], NULL, 0,\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> +\n> +\tswitch (discover_version(&reader)) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n> +\t\t\t\t &extra_have, &shallow);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> \n> \ttransport_verify_remote_names(nr_refspecs, refspecs);\n> \n> diff --git a/connect.c b/connect.c\n> index 00e90075c..db3c9d24c 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)\n> \t\t \"and the repository exists.\"));\n> }\n> \n> -static enum protocol_version discover_version(struct packet_reader *reader)\n> +enum protocol_version discover_version(struct packet_reader *reader)\n> {\n> \tenum protocol_version version = protocol_unknown_version;\n> \n> @@ -234,7 +234,7 @@ enum get_remote_heads_state {\n> /*\n> * Read all the refs from the other end\n> */\n> -struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> +struct ref **get_remote_heads(struct packet_reader *reader,\n> \t\t\t struct ref **list, unsigned int flags,\n> \t\t\t struct oid_array *extra_have,\n> \t\t\t struct oid_array *shallow_points)\n> @@ -242,24 +242,17 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \tstruct ref **orig_list = list;\n> \tint len = 0;\n> \tenum get_remote_heads_state state = EXPECTING_FIRST_REF;\n> -\tstruct packet_reader reader;\n> \tconst char *arg;\n> \n> -\tpacket_reader_init(&reader, in, src_buf, src_len,\n> -\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> -\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> -\n> -\tdiscover_version(&reader);\n> -\n> \t*list = NULL;\n> \n> \twhile (state != EXPECTING_DONE) {\n> -\t\tswitch (packet_reader_read(&reader)) {\n> +\t\tswitch (packet_reader_read(reader)) {\n> \t\tcase PACKET_READ_EOF:\n> \t\t\tdie_initial_contact(1);\n> \t\tcase PACKET_READ_NORMAL:\n> -\t\t\tlen = reader.pktlen;\n> -\t\t\tif (len > 4 && skip_prefix(reader.line, \"ERR \", &arg))\n> +\t\t\tlen = reader->pktlen;\n> +\t\t\tif (len > 4 && skip_prefix(reader->line, \"ERR \", &arg))\n> \t\t\t\tdie(\"remote error: %s\", arg);\n> \t\t\tbreak;\n> \t\tcase PACKET_READ_FLUSH:\n> @@ -271,22 +264,22 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \n> \t\tswitch (state) {\n> \t\tcase EXPECTING_FIRST_REF:\n> -\t\t\tprocess_capabilities(reader.line, &len);\n> -\t\t\tif (process_dummy_ref(reader.line)) {\n> +\t\t\tprocess_capabilities(reader->line, &len);\n> +\t\t\tif (process_dummy_ref(reader->line)) {\n> \t\t\t\tstate = EXPECTING_SHALLOW;\n> \t\t\t\tbreak;\n> \t\t\t}\n> \t\t\tstate = EXPECTING_REF;\n> \t\t\t/* fallthrough */\n> \t\tcase EXPECTING_REF:\n> -\t\t\tif (process_ref(reader.line, len, &list, flags, extra_have))\n> +\t\t\tif (process_ref(reader->line, len, &list, flags, extra_have))\n> \t\t\t\tbreak;\n> \t\t\tstate = EXPECTING_SHALLOW;\n> \t\t\t/* fallthrough */\n> \t\tcase EXPECTING_SHALLOW:\n> -\t\t\tif (process_shallow(reader.line, len, shallow_points))\n> +\t\t\tif (process_shallow(reader->line, len, shallow_points))\n> \t\t\t\tbreak;\n> -\t\t\tdie(\"protocol error: unexpected '%s'\", reader.line);\n> +\t\t\tdie(\"protocol error: unexpected '%s'\", reader->line);\n> \t\tcase EXPECTING_DONE:\n> \t\t\tbreak;\n> \t\t}\n> diff --git a/connect.h b/connect.h\n> index 01f14cdf3..cdb8979dc 100644\n> --- a/connect.h\n> +++ b/connect.h\n> @@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);\n> extern const char *server_feature_value(const char *feature, int *len_ret);\n> extern int url_is_local_not_ssh(const char *url);\n> \n> +struct packet_reader;\n> +extern enum protocol_version discover_version(struct packet_reader *reader);\n> +\n> #endif\n> diff --git a/remote-curl.c b/remote-curl.c\n> index 0053b0954..9f6d07683 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -1,6 +1,7 @@\n> #include \"cache.h\"\n> #include \"config.h\"\n> #include \"remote.h\"\n> +#include \"connect.h\"\n> #include \"strbuf.h\"\n> #include \"walker.h\"\n> #include \"http.h\"\n> @@ -13,6 +14,7 @@\n> #include \"credential.h\"\n> #include \"sha1-array.h\"\n> #include \"send-pack.h\"\n> +#include \"protocol.h\"\n> \n> static struct remote *remote;\n> /* always ends with a trailing slash */\n> @@ -176,8 +178,22 @@ static struct discovery *last_discovery;\n> static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n> {\n> \tstruct ref *list = NULL;\n> -\tget_remote_heads(-1, heads->buf, heads->len, &list,\n> -\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n> +\tstruct packet_reader reader;\n> +\n> +\tpacket_reader_init(&reader, -1, heads->buf, heads->len,\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> +\n> +\tswitch (discover_version(&reader)) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n> +\t\t\t\t NULL, &heads->shallow);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> +\n> \treturn list;\n> }\n> \n> diff --git a/remote.h b/remote.h\n> index 1f6611be2..2016461df 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n> void free_refs(struct ref *ref);\n> \n> struct oid_array;\n> -extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> +struct packet_reader;\n> +extern struct ref **get_remote_heads(struct packet_reader *reader,\n> \t\t\t\t struct ref **list, unsigned int flags,\n> \t\t\t\t struct oid_array *extra_have,\n> -\t\t\t\t struct oid_array *shallow);\n> +\t\t\t\t struct oid_array *shallow_points);\n> \n> int resolve_remote_symref(struct ref *ref, struct ref *list);\n> int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\n> diff --git a/transport.c b/transport.c\n> index 8e8779096..63c3dbab9 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -18,6 +18,7 @@\n> #include \"sha1-array.h\"\n> #include \"sigchain.h\"\n> #include \"transport-internal.h\"\n> +#include \"protocol.h\"\n> \n> static void set_upstreams(struct transport *transport, struct ref *refs,\n> \tint pretend)\n> @@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)\n> static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n> {\n> \tstruct git_transport_data *data = transport->data;\n> -\tstruct ref *refs;\n> +\tstruct ref *refs = NULL;\n> +\tstruct packet_reader reader;\n> \n> \tconnect_setup(transport, for_push);\n> -\tget_remote_heads(data->fd[0], NULL, 0, &refs,\n> -\t\t\t for_push ? REF_NORMAL : 0,\n> -\t\t\t &data->extra_have,\n> -\t\t\t &data->shallow);\n> +\n> +\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> +\n> +\tswitch (discover_version(&reader)) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tget_remote_heads(&reader, &refs,\n> +\t\t\t\t for_push ? REF_NORMAL : 0,\n> +\t\t\t\t &data->extra_have,\n> +\t\t\t\t &data->shallow);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> \tdata->got_remote_heads = 1;\n> \n> \treturn refs;\n\n"},{"id":"337958","messageId":"ae3bd6ae-7335-9aca-0818-a90e202ff37a@gmail.com","threadId":"47524","inReplyTo":"20180125235838.138135-10-bmwill@google.com","subject":"Re: [PATCH v2 09/27] transport: store protocol version","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-31T14:45:43Z","receivedAt":"2018-01-31T14:45:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/25/2018 6:58 PM, Brandon Williams wrote:\n> +\tswitch (data->version) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\trefs = fetch_pack(&args, data->fd, data->conn,\n> +\t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n> +\t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n> +\t\t\t\t &transport->pack_lockfile);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n\nAfter seeing this pattern a few times, I think it would be good to \nconvert it to a macro that calls a statement for protocol_v1/v0 (and \nlater calls a different one for protocol_v2). It would at minimum reduce \nthe code clones surrounding this handling of unknown_version, and we \ncould have one place that is clear this BUG() is due to an unexpected \nresponse from discover_version().\n\n"},{"id":"337959","messageId":"815183f7-f449-1286-6dee-7cf6064bc5a1@gmail.com","threadId":"47524","inReplyTo":"20180125235838.138135-11-bmwill@google.com","subject":"Re: [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-31T14:54:41Z","receivedAt":"2018-01-31T14:54:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/25/2018 6:58 PM, Brandon Williams wrote:\n> Introduce protocol_v2, a new value for 'enum protocol_version'.\n> Subsequent patches will fill in the implementation of protocol_v2.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> builtin/fetch-pack.c | 3 +++\n> builtin/receive-pack.c | 6 ++++++\n> builtin/send-pack.c | 3 +++\n> builtin/upload-pack.c | 7 +++++++\n> connect.c | 3 +++\n> protocol.c | 2 ++\n> protocol.h | 1 +\n> remote-curl.c | 3 +++\n> transport.c | 9 +++++++++\n> 9 files changed, 37 insertions(+)\n>\n> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n> index 85d4faf76..f492e8abd 100644\n> --- a/builtin/fetch-pack.c\n> +++ b/builtin/fetch-pack.c\n> @@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> \t\t\t PACKET_READ_GENTLE_ON_EOF);\n> \n> \tswitch (discover_version(&reader)) {\n> +\tcase protocol_v2:\n> +\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tbreak;\n> \tcase protocol_v1:\n> \tcase protocol_v0:\n> \t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> index b7ce7c7f5..3656e94fd 100644\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> \t\tunpack_limit = receive_unpack_limit;\n> \n> \tswitch (determine_protocol_version_server()) {\n> +\tcase protocol_v2:\n> +\t\t/*\n> +\t\t * push support for protocol v2 has not been implemented yet,\n> +\t\t * so ignore the request to use v2 and fallback to using v0.\n> +\t\t */\n> +\t\tbreak;\n> \tcase protocol_v1:\n> \t\t/*\n> \t\t * v1 is just the original protocol with a version string,\n> diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n> index 83cb125a6..b5427f75e 100644\n> --- a/builtin/send-pack.c\n> +++ b/builtin/send-pack.c\n> @@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n> \t\t\t PACKET_READ_GENTLE_ON_EOF);\n> \n> \tswitch (discover_version(&reader)) {\n> +\tcase protocol_v2:\n> +\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tbreak;\n> \tcase protocol_v1:\n> \tcase protocol_v0:\n> \t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n> diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\n> index 2cb5cb35b..8d53e9794 100644\n> --- a/builtin/upload-pack.c\n> +++ b/builtin/upload-pack.c\n> @@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n> \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n> \n> \tswitch (determine_protocol_version_server()) {\n> +\tcase protocol_v2:\n> +\t\t/*\n> +\t\t * fetch support for protocol v2 has not been implemented yet,\n> +\t\t * so ignore the request to use v2 and fallback to using v0.\n> +\t\t */\n> +\t\tupload_pack(&opts);\n> +\t\tbreak;\n> \tcase protocol_v1:\n> \t\t/*\n> \t\t * v1 is just the original protocol with a version string,\n> diff --git a/connect.c b/connect.c\n> index db3c9d24c..f2157a821 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)\n> \n> \t/* Maybe process capabilities here, at least for v2 */\n> \tswitch (version) {\n> +\tcase protocol_v2:\n> +\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tbreak;\n> \tcase protocol_v1:\n> \t\t/* Read the peeked version line */\n> \t\tpacket_reader_read(reader);\n> diff --git a/protocol.c b/protocol.c\n> index 43012b7eb..5e636785d 100644\n> --- a/protocol.c\n> +++ b/protocol.c\n> @@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)\n> \t\treturn protocol_v0;\n> \telse if (!strcmp(value, \"1\"))\n> \t\treturn protocol_v1;\n> +\telse if (!strcmp(value, \"2\"))\n> +\t\treturn protocol_v2;\n> \telse\n> \t\treturn protocol_unknown_version;\n> }\n> diff --git a/protocol.h b/protocol.h\n> index 1b2bc94a8..2ad35e433 100644\n> --- a/protocol.h\n> +++ b/protocol.h\n> @@ -5,6 +5,7 @@ enum protocol_version {\n> \tprotocol_unknown_version = -1,\n> \tprotocol_v0 = 0,\n> \tprotocol_v1 = 1,\n> +\tprotocol_v2 = 2,\n> };\n> \n> /*\n> diff --git a/remote-curl.c b/remote-curl.c\n> index 9f6d07683..dae8a4a48 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n> \t\t\t PACKET_READ_GENTLE_ON_EOF);\n> \n> \tswitch (discover_version(&reader)) {\n> +\tcase protocol_v2:\n> +\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tbreak;\n> \tcase protocol_v1:\n> \tcase protocol_v0:\n> \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n> diff --git a/transport.c b/transport.c\n> index 2378dcb38..83d9dd1df 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n> \n> \tdata->version = discover_version(&reader);\n> \tswitch (data->version) {\n> +\tcase protocol_v2:\n> +\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tbreak;\n> \tcase protocol_v1:\n> \tcase protocol_v0:\n> \t\tget_remote_heads(&reader, &refs,\n> @@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \t\trefs_tmp = get_refs_via_connect(transport, 0);\n> \n> \tswitch (data->version) {\n> +\tcase protocol_v2:\n> +\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tbreak;\n> \tcase protocol_v1:\n> \tcase protocol_v0:\n> \t\trefs = fetch_pack(&args, data->fd, data->conn,\n> @@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n> \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n> \n> \tswitch (data->version) {\n> +\tcase protocol_v2:\n> +\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tbreak;\n> \tcase protocol_v1:\n> \tcase protocol_v0:\n> \t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n\nWith a macro approach to version selection, this change becomes simpler \nin some ways and harder in others.\n\nIt is simpler in that we can have the macro from the previous commits \njust fall back to version 0 behavior.\n\nIt is harder in that this commit would need one of two options:\n\n1. A macro that performs an arbitrary statement when given v2, which \nwould be the die() for these actions not in v2.\n2. A macro that clearly states v2 is not supported and calls die() on v2.\n\nHere is my simple, untested attempt at a union of these options:\n\n#define ON_PROTOCOL_VERSION(version,v0,v2) switch(version) {\\\ncase protocol_v2:\\\n     (v2);\\\n     break;\\\ncase protocol_v1:\\\ncase protocol_v0:\\\n     (v0);\\\n     break;\\\ncase protocol_unknown_version:\\\n     BUG(\"unknown protocol version\");\\\n}\n#define ON_PROTOCOL_VERSION_V0_FALLBACK(version,v0) switch(version) {\\\ncase protocol_v2:\\\ncase protocol_v1:\\\ncase protocol_v0:\\\n     (v0);\\\n     break;\\\ncase protocol_unknown_version:\\\n     BUG(\"unknown protocol version\");\\\n}\n#define ON_PROTOCOL_VERSION_V0_ONLY(version,v0) \\\n     ON_PROTOCOL_VERSION(version,v0,\\\n                 BUG(\"support for protocol v2 not implemented yet\"))\n"},{"id":"337961","messageId":"2007efc8-ec31-0267-9b83-392a9d2eaf4a@gmail.com","threadId":"47524","inReplyTo":"20180125235838.138135-15-bmwill@google.com","subject":"Re: [PATCH v2 14/27] connect: request remote refs using v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-31T15:22:46Z","receivedAt":"2018-01-31T15:22:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/25/2018 6:58 PM, Brandon Williams wrote:\n> Teach the client to be able to request a remote's refs using protocol\n> v2. This is done by having a client issue a 'ls-refs' request to a v2\n> server.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> builtin/upload-pack.c | 10 ++--\n> connect.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++-\n> remote.h | 4 ++\n> t/t5702-protocol-v2.sh | 28 +++++++++++\n> transport.c | 2 +-\n> 5 files changed, 160 insertions(+), 7 deletions(-)\n> create mode 100755 t/t5702-protocol-v2.sh\n>\n> diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\n> index 8d53e9794..a757df8da 100644\n> --- a/builtin/upload-pack.c\n> +++ b/builtin/upload-pack.c\n> @@ -5,6 +5,7 @@\n> #include \"parse-options.h\"\n> #include \"protocol.h\"\n> #include \"upload-pack.h\"\n> +#include \"serve.h\"\n> \n> static const char * const upload_pack_usage[] = {\n> \tN_(\"git upload-pack [] \"),\n> @@ -16,6 +17,7 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n> \tconst char *dir;\n> \tint strict = 0;\n> \tstruct upload_pack_options opts = { 0 };\n> +\tstruct serve_options serve_opts = SERVE_OPTIONS_INIT;\n> \tstruct option options[] = {\n> \t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n> \t\t\t N_(\"quit after a single request/response exchange\")),\n> @@ -48,11 +50,9 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n> \n> \tswitch (determine_protocol_version_server()) {\n> \tcase protocol_v2:\n> -\t\t/*\n> -\t\t * fetch support for protocol v2 has not been implemented yet,\n> -\t\t * so ignore the request to use v2 and fallback to using v0.\n> -\t\t */\n> -\t\tupload_pack(&opts);\n> +\t\tserve_opts.advertise_capabilities = opts.advertise_refs;\n> +\t\tserve_opts.stateless_rpc = opts.stateless_rpc;\n> +\t\tserve(&serve_opts);\n> \t\tbreak;\n> \tcase protocol_v1:\n> \t\t/*\n> diff --git a/connect.c b/connect.c\n> index f2157a821..3c653b65b 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -12,9 +12,11 @@\n> #include \"sha1-array.h\"\n> #include \"transport.h\"\n> #include \"strbuf.h\"\n> +#include \"version.h\"\n> #include \"protocol.h\"\n> \n> static char *server_capabilities;\n> +static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n> static const char *parse_feature_value(const char *, const char *, int *);\n> \n> static int check_ref(const char *name, unsigned int flags)\n> @@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)\n> \t\t \"and the repository exists.\"));\n> }\n> \n> +/* Checks if the server supports the capability 'c' */\n> +static int server_supports_v2(const char *c, int die_on_error)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n> +\t\tconst char *out;\n> +\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n> +\t\t (!*out || *out == '='))\n> +\t\t\treturn 1;\n> +\t}\n> +\n> +\tif (die_on_error)\n> +\t\tdie(\"server doesn't support '%s'\", c);\n> +\n> +\treturn 0;\n> +}\n> +\n> +static void process_capabilities_v2(struct packet_reader *reader)\n> +{\n> +\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL)\n> +\t\targv_array_push(&server_capabilities_v2, reader->line);\n> +\n> +\tif (reader->status != PACKET_READ_FLUSH)\n> +\t\tdie(\"protocol error\");\n> +}\n> +\n> enum protocol_version discover_version(struct packet_reader *reader)\n> {\n> \tenum protocol_version version = protocol_unknown_version;\n> @@ -85,7 +114,7 @@ enum protocol_version discover_version(struct packet_reader *reader)\n> \t/* Maybe process capabilities here, at least for v2 */\n> \tswitch (version) {\n> \tcase protocol_v2:\n> -\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tprocess_capabilities_v2(reader);\n> \t\tbreak;\n> \tcase protocol_v1:\n> \t\t/* Read the peeked version line */\n> @@ -293,6 +322,98 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n> \treturn list;\n> }\n> \n> +static int process_ref_v2(const char *line, struct ref ***list)\n> +{\n> +\tint ret = 1;\n> +\tint i = 0;\n\nnit: you set 'i' here, but first use it in a for loop with blank \ninitializer. Perhaps keep the first assignment closer to the first use?\n\n> +\tstruct object_id old_oid;\n> +\tstruct ref *ref;\n> +\tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n> +\n> +\tif (string_list_split(&line_sections, line, ' ', -1) < 2) {\n> +\t\tret = 0;\n> +\t\tgoto out;\n> +\t}\n> +\n> +\tif (get_oid_hex(line_sections.items[i++].string, &old_oid)) {\n> +\t\tret = 0;\n> +\t\tgoto out;\n> +\t}\n> +\n> +\tref = alloc_ref(line_sections.items[i++].string);\n> +\n> +\toidcpy(&ref->old_oid, &old_oid);\n> +\t**list = ref;\n> +\t*list = &ref->next;\n> +\n> +\tfor (; i < line_sections.nr; i++) {\n> +\t\tconst char *arg = line_sections.items[i].string;\n> +\t\tif (skip_prefix(arg, \"symref-target:\", &arg))\n> +\t\t\tref->symref = xstrdup(arg);\n> +\n> +\t\tif (skip_prefix(arg, \"peeled:\", &arg)) {\n> +\t\t\tstruct object_id peeled_oid;\n> +\t\t\tchar *peeled_name;\n> +\t\t\tstruct ref *peeled;\n> +\t\t\tif (get_oid_hex(arg, &peeled_oid)) {\n> +\t\t\t\tret = 0;\n> +\t\t\t\tgoto out;\n> +\t\t\t}\n> +\n> +\t\t\tpeeled_name = xstrfmt(\"%s^{}\", ref->name);\n> +\t\t\tpeeled = alloc_ref(peeled_name);\n> +\n> +\t\t\toidcpy(&peeled->old_oid, &peeled_oid);\n> +\t\t\t**list = peeled;\n> +\t\t\t*list = &peeled->next;\n> +\n> +\t\t\tfree(peeled_name);\n> +\t\t}\n> +\t}\n> +\n> +out:\n> +\tstring_list_clear(&line_sections, 0);\n> +\treturn ret;\n> +}\n> +\n> +struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> +\t\t\t struct ref **list, int for_push,\n> +\t\t\t const struct argv_array *ref_patterns)\n> +{\n> +\tint i;\n> +\t*list = NULL;\n> +\n> +\t/* Check that the server supports the ls-refs command */\n> +\t/* Issue request for ls-refs */\n> +\tif (server_supports_v2(\"ls-refs\", 1))\n> +\t\tpacket_write_fmt(fd_out, \"command=ls-refs\\n\");\n> +\n> +\tif (server_supports_v2(\"agent\", 0))\n> +\t packet_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n> +\n> +\tpacket_delim(fd_out);\n> +\t/* When pushing we don't want to request the peeled tags */\n> +\tif (!for_push)\n> +\t\tpacket_write_fmt(fd_out, \"peel\\n\");\n> +\tpacket_write_fmt(fd_out, \"symrefs\\n\");\n> +\tfor (i = 0; ref_patterns && i < ref_patterns->argc; i++) {\n> +\t\tpacket_write_fmt(fd_out, \"ref-pattern %s\\n\",\n> +\t\t\t\t ref_patterns->argv[i]);\n> +\t}\n> +\tpacket_flush(fd_out);\n> +\n> +\t/* Process response from server */\n> +\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n> +\t\tif (!process_ref_v2(reader->line, &list))\n> +\t\t\tdie(\"invalid ls-refs response: %s\", reader->line);\n> +\t}\n> +\n> +\tif (reader->status != PACKET_READ_FLUSH)\n> +\t\tdie(\"protocol error\");\n> +\n> +\treturn list;\n> +}\n> +\n> static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)\n> {\n> \tint len;\n> diff --git a/remote.h b/remote.h\n> index 2016461df..21d0c776c 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -151,10 +151,14 @@ void free_refs(struct ref *ref);\n> \n> struct oid_array;\n> struct packet_reader;\n> +struct argv_array;\n> extern struct ref **get_remote_heads(struct packet_reader *reader,\n> \t\t\t\t struct ref **list, unsigned int flags,\n> \t\t\t\t struct oid_array *extra_have,\n> \t\t\t\t struct oid_array *shallow_points);\n> +extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> +\t\t\t\t struct ref **list, int for_push,\n> +\t\t\t\t const struct argv_array *ref_patterns);\n> \n> int resolve_remote_symref(struct ref *ref, struct ref *list);\n> int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\n> diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\n> new file mode 100755\n> index 000000000..4bf4d61ac\n> --- /dev/null\n> +++ b/t/t5702-protocol-v2.sh\n> @@ -0,0 +1,28 @@\n> +#!/bin/sh\n> +\n> +test_description='test git wire-protocol version 2'\n> +\n> +TEST_NO_CREATE_REPO=1\n> +\n> +. ./test-lib.sh\n> +\n> +# Test protocol v2 with 'file://' transport\n> +#\n> +test_expect_success 'create repo to be served by file:// transport' '\n> +\tgit init file_parent &&\n> +\ttest_commit -C file_parent one\n> +'\n> +\n> +test_expect_success 'list refs with file:// using protocol v2' '\n> +\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n> +\t\tls-remote --symref \"file://$(pwd)/file_parent\" >actual 2>log &&\n> +\n> +\t# Server responded using protocol v2\n> +\tcat log &&\n> +\tgrep \"git< version 2\" log &&\n> +\n> +\tgit ls-remote --symref \"file://$(pwd)/file_parent\" >expect &&\n> +\ttest_cmp actual expect\n> +'\n> +\n> +test_done\n> diff --git a/transport.c b/transport.c\n> index 83d9dd1df..ffc6b2614 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -204,7 +204,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n> \tdata->version = discover_version(&reader);\n> \tswitch (data->version) {\n> \tcase protocol_v2:\n> -\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);\n> \t\tbreak;\n> \tcase protocol_v1:\n> \tcase protocol_v0:\n\n"},{"id":"337962","messageId":"a9500d32-3f0a-4979-dcde-fb2acf0b3434@gmail.com","threadId":"47524","inReplyTo":"20180125235838.138135-13-bmwill@google.com","subject":"Re: [PATCH v2 12/27] serve: introduce git-serve","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-31T15:39:59Z","receivedAt":"2018-01-31T15:40:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/25/2018 6:58 PM, Brandon Williams wrote:\n> Introduce git-serve, the base server for protocol version 2.\n>\n> Protocol version 2 is intended to be a replacement for Git's current\n> wire protocol. The intention is that it will be a simpler, less\n> wasteful protocol which can evolve over time.\n>\n> Protocol version 2 improves upon version 1 by eliminating the initial\n> ref advertisement. In its place a server will export a list of\n> capabilities and commands which it supports in a capability\n> advertisement. A client can then request that a particular command be\n> executed by providing a number of capabilities and command specific\n> parameters. At the completion of a command, a client can request that\n> another command be executed or can terminate the connection by sending a\n> flush packet.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> .gitignore | 1 +\n> Documentation/technical/protocol-v2.txt | 117 +++++++++++++++\n> Makefile | 2 +\n> builtin.h | 1 +\n> builtin/serve.c | 30 ++++\n> git.c | 1 +\n> serve.c | 249 ++++++++++++++++++++++++++++++++\n> serve.h | 15 ++\n> t/t5701-git-serve.sh | 56 +++++++\n> 9 files changed, 472 insertions(+)\n> create mode 100644 Documentation/technical/protocol-v2.txt\n> create mode 100644 builtin/serve.c\n> create mode 100644 serve.c\n> create mode 100644 serve.h\n> create mode 100755 t/t5701-git-serve.sh\n>\n> diff --git a/.gitignore b/.gitignore\n> index 833ef3b0b..2d0450c26 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -140,6 +140,7 @@\n> /git-rm\n> /git-send-email\n> /git-send-pack\n> +/git-serve\n> /git-sh-i18n\n> /git-sh-i18n--envsubst\n> /git-sh-setup\n> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> new file mode 100644\n> index 000000000..7f619a76c\n> --- /dev/null\n> +++ b/Documentation/technical/protocol-v2.txt\n> @@ -0,0 +1,117 @@\n> + Git Wire Protocol, Version 2\n> +==============================\n> +\n> +This document presents a specification for a version 2 of Git's wire\n> +protocol. Protocol v2 will improve upon v1 in the following ways:\n> +\n> + * Instead of multiple service names, multiple commands will be\n> + supported by a single service.\n\nAs someone unfamiliar with the old protocol code, this statement is \nunderselling the architectural significance of your change. The new \nmodel allows a single service to handle all different wire protocols \n(git://, ssh://, https://) while being agnostic to the command-specific \nlogic. It also hides the protocol negotiation away from these consumers.\n\nThe ease with which you are adding new commands in later commits really \ndemonstrates the value of this patch. To make that point here, you would \nalmost need to document the old model to show how it was difficult to \nuse and extend. Perhaps this document will not need expanding since the \ncode speaks for itself.\n\nI just wanted to state for the record that the new architecture is a big \nimprovement and will make more commands much easier to implement.\n\n> + * Easily extendable as capabilities are moved into their own section\n> + of the protocol, no longer being hidden behind a NUL byte and\n> + limited by the size of a pkt-line (as there will be a single\n> + capability per pkt-line).\n> + * Separate out other information hidden behind NUL bytes (e.g. agent\n> + string as a capability and symrefs can be requested using 'ls-refs')\n> + * Reference advertisement will be omitted unless explicitly requested\n> + * ls-refs command to explicitly request some refs\n> +\n\nnit: some bullets have full stops (.) and others do not.\n\n> + Detailed Design\n> +=================\n> +\n> +A client can request to speak protocol v2 by sending `version=2` in the\n> +side-channel `GIT_PROTOCOL` in the initial request to the server.\n> +\n> +In protocol v2 communication is command oriented. When first contacting a\n> +server a list of capabilities will advertised. Some of these capabilities\n> +will be commands which a client can request be executed. Once a command\n> +has completed, a client can reuse the connection and request that other\n> +commands be executed.\n> +\n> + Special Packets\n> +-----------------\n> +\n> +In protocol v2 these special packets will have the following semantics:\n> +\n> + * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n> + * '0001' Delimiter Packet (delim-pkt) - separates sections of a message\n> +\n> + Capability Advertisement\n> +--------------------------\n> +\n> +A server which decides to communicate (based on a request from a client)\n> +using protocol version 2, notifies the client by sending a version string\n> +in its initial response followed by an advertisement of its capabilities.\n> +Each capability is a key with an optional value. Clients must ignore all\n> +unknown keys. Semantics of unknown values are left to the definition of\n> +each key. Some capabilities will describe commands which can be requested\n> +to be executed by the client.\n> +\n> + capability-advertisement = protocol-version\n> +\t\t\t capability-list\n> +\t\t\t flush-pkt\n> +\n> + protocol-version = PKT-LINE(\"version 2\" LF)\n> + capability-list = *capability\n> + capability = PKT-LINE(key[=value] LF)\n> +\n> + key = 1*CHAR\n> + value = 1*CHAR\n> + CHAR = 1*(ALPHA / DIGIT / \"-\" / \"_\")\n> +\n> +A client then responds to select the command it wants with any particular\n> +capabilities or arguments. There is then an optional section where the\n> +client can provide any command specific parameters or queries.\n> +\n> + command-request = command\n> +\t\t capability-list\n> +\t\t (command-args)\n> +\t\t flush-pkt\n> + command = PKT-LINE(\"command=\" key LF)\n> + command-args = delim-pkt\n> +\t\t *arg\n> + arg = 1*CHAR\n> +\n> +The server will then check to ensure that the client's request is\n> +comprised of a valid command as well as valid capabilities which were\n> +advertised. If the request is valid the server will then execute the\n> +command.\n> +\n> +When a command has finished a client can either request that another\n> +command be executed or can terminate the connection by sending an empty\n> +request consisting of just a flush-pkt.\n> +\n> + Capabilities\n> +~~~~~~~~~~~~~~\n> +\n> +There are two different types of capabilities: normal capabilities,\n> +which can be used to to convey information or alter the behavior of a\n> +request, and command capabilities, which are the core actions that a\n> +client wants to perform (fetch, push, etc).\n> +\n> + agent\n> +-------\n> +\n> +The server can advertise the `agent` capability with a value `X` (in the\n> +form `agent=X`) to notify the client that the server is running version\n> +`X`. The client may optionally send its own agent string by including\n> +the `agent` capability with a value `Y` (in the form `agent=Y`) in its\n> +request to the server (but it MUST NOT do so if the server did not\n> +advertise the agent capability). The `X` and `Y` strings may contain any\n> +printable ASCII characters except space (i.e., the byte range 32 < x <\n> +127), and are typically of the form \"package/version\" (e.g.,\n> +\"git/1.8.3.1\"). The agent strings are purely informative for statistics\n> +and debugging purposes, and MUST NOT be used to programmatically assume\n> +the presence or absence of particular features.\n> +\n> + stateless-rpc\n> +---------------\n> +\n> +If advertised, the `stateless-rpc` capability indicates that the server\n> +supports running commands in a stateless-rpc mode, which means that a\n> +command lasts for only a single request-response round.\n> +\n> +Normally a command can last for as many rounds as are required to\n> +complete it (multiple for negotiation during fetch or no additional\n> +trips in the case of ls-refs). If the client sends the `stateless-rpc`\n> +capability with a value of `true` (in the form `stateless-rpc=true`)\n> +then the invoked command must only last a single round.\n> diff --git a/Makefile b/Makefile\n> index 3b849c060..18c255428 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -881,6 +881,7 @@ LIB_OBJS += revision.o\n> LIB_OBJS += run-command.o\n> LIB_OBJS += send-pack.o\n> LIB_OBJS += sequencer.o\n> +LIB_OBJS += serve.o\n> LIB_OBJS += server-info.o\n> LIB_OBJS += setup.o\n> LIB_OBJS += sha1-array.o\n> @@ -1014,6 +1015,7 @@ BUILTIN_OBJS += builtin/rev-parse.o\n> BUILTIN_OBJS += builtin/revert.o\n> BUILTIN_OBJS += builtin/rm.o\n> BUILTIN_OBJS += builtin/send-pack.o\n> +BUILTIN_OBJS += builtin/serve.o\n> BUILTIN_OBJS += builtin/shortlog.o\n> BUILTIN_OBJS += builtin/show-branch.o\n> BUILTIN_OBJS += builtin/show-ref.o\n> diff --git a/builtin.h b/builtin.h\n> index f332a1257..3f3fdfc28 100644\n> --- a/builtin.h\n> +++ b/builtin.h\n> @@ -215,6 +215,7 @@ extern int cmd_rev_parse(int argc, const char **argv, const char *prefix);\n> extern int cmd_revert(int argc, const char **argv, const char *prefix);\n> extern int cmd_rm(int argc, const char **argv, const char *prefix);\n> extern int cmd_send_pack(int argc, const char **argv, const char *prefix);\n> +extern int cmd_serve(int argc, const char **argv, const char *prefix);\n> extern int cmd_shortlog(int argc, const char **argv, const char *prefix);\n> extern int cmd_show(int argc, const char **argv, const char *prefix);\n> extern int cmd_show_branch(int argc, const char **argv, const char *prefix);\n> diff --git a/builtin/serve.c b/builtin/serve.c\n> new file mode 100644\n> index 000000000..d3fd240bb\n> --- /dev/null\n> +++ b/builtin/serve.c\n> @@ -0,0 +1,30 @@\n> +#include \"cache.h\"\n> +#include \"builtin.h\"\n> +#include \"parse-options.h\"\n> +#include \"serve.h\"\n> +\n> +static char const * const serve_usage[] = {\n> +\tN_(\"git serve []\"),\n> +\tNULL\n> +};\n> +\n> +int cmd_serve(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstruct serve_options opts = SERVE_OPTIONS_INIT;\n> +\n> +\tstruct option options[] = {\n> +\t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n> +\t\t\t N_(\"quit after a single request/response exchange\")),\n> +\t\tOPT_BOOL(0, \"advertise-capabilities\", &opts.advertise_capabilities,\n> +\t\t\t N_(\"exit immediately after advertising capabilities\")),\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\t/* ignore all unknown cmdline switches for now */\n> +\targc = parse_options(argc, argv, prefix, options, serve_usage,\n> +\t\t\t PARSE_OPT_KEEP_DASHDASH |\n> +\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n> +\tserve(&opts);\n> +\n> +\treturn 0;\n> +}\n> diff --git a/git.c b/git.c\n> index f71073dc8..f85d682b6 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -461,6 +461,7 @@ static struct cmd_struct commands[] = {\n> \t{ \"revert\", cmd_revert, RUN_SETUP | NEED_WORK_TREE },\n> \t{ \"rm\", cmd_rm, RUN_SETUP },\n> \t{ \"send-pack\", cmd_send_pack, RUN_SETUP },\n> +\t{ \"serve\", cmd_serve, RUN_SETUP },\n> \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n> \t{ \"show\", cmd_show, RUN_SETUP },\n> \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n> diff --git a/serve.c b/serve.c\n> new file mode 100644\n> index 000000000..90e3defe8\n> --- /dev/null\n> +++ b/serve.c\n> @@ -0,0 +1,249 @@\n> +#include \"cache.h\"\n> +#include \"repository.h\"\n> +#include \"config.h\"\n> +#include \"pkt-line.h\"\n> +#include \"version.h\"\n> +#include \"argv-array.h\"\n> +#include \"serve.h\"\n> +\n> +static int always_advertise(struct repository *r,\n> +\t\t\t struct strbuf *value)\n> +{\n> +\treturn 1;\n> +}\n> +\n> +static int agent_advertise(struct repository *r,\n> +\t\t\t struct strbuf *value)\n> +{\n> +\tif (value)\n> +\t\tstrbuf_addstr(value, git_user_agent_sanitized());\n> +\treturn 1;\n> +}\n> +\n> +struct protocol_capability {\n> +\t/*\n> +\t * The name of the capability. The server uses this name when\n> +\t * advertising this capability, and the client uses this name to\n> +\t * specify this capability.\n> +\t */\n> +\tconst char *name;\n> +\n> +\t/*\n> +\t * Function queried to see if a capability should be advertised.\n> +\t * Optionally a value can be specified by adding it to 'value'.\n> +\t * If a value is added to 'value', the server will advertise this\n> +\t * capability as \"=\" instead of \"\".\n> +\t */\n> +\tint (*advertise)(struct repository *r, struct strbuf *value);\n> +\n> +\t/*\n> +\t * Function called when a client requests the capability as a command.\n> +\t * The command request will be provided to the function via 'keys', the\n> +\t * capabilities requested, and 'args', the command specific parameters.\n> +\t *\n> +\t * This field should be NULL for capabilities which are not commands.\n> +\t */\n> +\tint (*command)(struct repository *r,\n> +\t\t struct argv_array *keys,\n> +\t\t struct argv_array *args);\n> +};\n> +\n> +static struct protocol_capability capabilities[] = {\n> +\t{ \"agent\", agent_advertise, NULL },\n> +\t{ \"stateless-rpc\", always_advertise, NULL },\n> +};\n> +\n> +static void advertise_capabilities(void)\n> +{\n> +\tstruct strbuf capability = STRBUF_INIT;\n> +\tstruct strbuf value = STRBUF_INIT;\n> +\tint i;\n> +\n> +\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n> +\t\tstruct protocol_capability *c = &capabilities[i];\n> +\n> +\t\tif (c->advertise(the_repository, &value)) {\n> +\t\t\tstrbuf_addstr(&capability, c->name);\n> +\n> +\t\t\tif (value.len) {\n> +\t\t\t\tstrbuf_addch(&capability, '=');\n> +\t\t\t\tstrbuf_addbuf(&capability, &value);\n> +\t\t\t}\n> +\n> +\t\t\tstrbuf_addch(&capability, '\\n');\n> +\t\t\tpacket_write(1, capability.buf, capability.len);\n> +\t\t}\n> +\n> +\t\tstrbuf_reset(&capability);\n> +\t\tstrbuf_reset(&value);\n> +\t}\n> +\n> +\tpacket_flush(1);\n> +\tstrbuf_release(&capability);\n> +\tstrbuf_release(&value);\n> +}\n> +\n> +static struct protocol_capability *get_capability(const char *key)\n> +{\n> +\tint i;\n> +\n> +\tif (!key)\n> +\t\treturn NULL;\n> +\n> +\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n> +\t\tstruct protocol_capability *c = &capabilities[i];\n> +\t\tconst char *out;\n> +\t\tif (skip_prefix(key, c->name, &out) && (!*out || *out == '='))\n> +\t\t\treturn c;\n> +\t}\n> +\n> +\treturn NULL;\n> +}\n> +\n> +static int is_valid_capability(const char *key)\n> +{\n> +\tconst struct protocol_capability *c = get_capability(key);\n> +\n> +\treturn c && c->advertise(the_repository, NULL);\n> +}\n> +\n> +static int is_command(const char *key, struct protocol_capability **command)\n> +{\n> +\tconst char *out;\n> +\n> +\tif (skip_prefix(key, \"command=\", &out)) {\n> +\t\tstruct protocol_capability *cmd = get_capability(out);\n> +\n> +\t\tif (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)\n> +\t\t\tdie(\"invalid command '%s'\", out);\n> +\t\tif (*command)\n> +\t\t\tdie(\"command already requested\");\n> +\n> +\t\t*command = cmd;\n> +\t\treturn 1;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +int has_capability(const struct argv_array *keys, const char *capability,\n> +\t\t const char **value)\n> +{\n> +\tint i;\n> +\tfor (i = 0; i < keys->argc; i++) {\n> +\t\tconst char *out;\n> +\t\tif (skip_prefix(keys->argv[i], capability, &out) &&\n> +\t\t (!*out || *out == '=')) {\n> +\t\t\tif (value) {\n> +\t\t\t\tif (*out == '=')\n> +\t\t\t\t\tout++;\n> +\t\t\t\t*value = out;\n> +\t\t\t}\n> +\t\t\treturn 1;\n> +\t\t}\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +enum request_state {\n> +\tPROCESS_REQUEST_KEYS = 0,\n> +\tPROCESS_REQUEST_ARGS,\n> +\tPROCESS_REQUEST_DONE,\n> +};\n> +\n> +static int process_request(void)\n> +{\n> +\tenum request_state state = PROCESS_REQUEST_KEYS;\n> +\tchar *buffer = packet_buffer;\n> +\tunsigned buffer_size = sizeof(packet_buffer);\n> +\tint pktlen;\n> +\tstruct argv_array keys = ARGV_ARRAY_INIT;\n> +\tstruct argv_array args = ARGV_ARRAY_INIT;\n> +\tstruct protocol_capability *command = NULL;\n> +\n> +\twhile (state != PROCESS_REQUEST_DONE) {\n> +\t\tswitch (packet_read_with_status(0, NULL, NULL, buffer,\n> +\t\t\t\t\t\tbuffer_size, &pktlen,\n> +\t\t\t\t\t\tPACKET_READ_CHOMP_NEWLINE)) {\n> +\t\tcase PACKET_READ_EOF:\n> +\t\t\tBUG(\"Should have already died when seeing EOF\");\n> +\t\tcase PACKET_READ_NORMAL:\n> +\t\t\tbreak;\n> +\t\tcase PACKET_READ_FLUSH:\n> +\t\t\tstate = PROCESS_REQUEST_DONE;\n> +\t\t\tcontinue;\n> +\t\tcase PACKET_READ_DELIM:\n> +\t\t\tif (state != PROCESS_REQUEST_KEYS)\n> +\t\t\t\tdie(\"protocol error\");\n> +\t\t\tstate = PROCESS_REQUEST_ARGS;\n> +\t\t\t/*\n> +\t\t\t * maybe include a check to make sure that a\n> +\t\t\t * command/capabilities were given.\n> +\t\t\t */\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tswitch (state) {\n> +\t\tcase PROCESS_REQUEST_KEYS:\n> +\t\t\t/* collect request; a sequence of keys and values */\n> +\t\t\tif (is_command(buffer, &command) ||\n> +\t\t\t is_valid_capability(buffer))\n> +\t\t\t\targv_array_push(&keys, buffer);\n> +\t\t\telse\n> +\t\t\t\tdie(\"unknown capability '%s'\", buffer);\n> +\t\t\tbreak;\n> +\t\tcase PROCESS_REQUEST_ARGS:\n> +\t\t\t/* collect arguments for the requested command */\n> +\t\t\targv_array_push(&args, buffer);\n> +\t\t\tbreak;\n> +\t\tcase PROCESS_REQUEST_DONE:\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * If no command and no keys were given then the client wanted to\n> +\t * terminate the connection.\n> +\t */\n> +\tif (!keys.argc && !args.argc)\n> +\t\treturn 1;\n> +\n> +\tif (!command)\n> +\t\tdie(\"no command requested\");\n> +\n> +\tcommand->command(the_repository, &keys, &args);\n> +\n> +\targv_array_clear(&keys);\n> +\targv_array_clear(&args);\n> +\treturn 0;\n> +}\n> +\n> +/* Main serve loop for protocol version 2 */\n> +void serve(struct serve_options *options)\n> +{\n> +\tif (options->advertise_capabilities || !options->stateless_rpc) {\n> +\t\t/* serve by default supports v2 */\n> +\t\tpacket_write_fmt(1, \"version 2\\n\");\n> +\n> +\t\tadvertise_capabilities();\n> +\t\t/*\n> +\t\t * If only the list of capabilities was requested exit\n> +\t\t * immediately after advertising capabilities\n> +\t\t */\n> +\t\tif (options->advertise_capabilities)\n> +\t\t\treturn;\n> +\t}\n> +\n> +\t/*\n> +\t * If stateless-rpc was requested then exit after\n> +\t * a single request/response exchange\n> +\t */\n> +\tif (options->stateless_rpc) {\n> +\t\tprocess_request();\n> +\t} else {\n> +\t\tfor (;;)\n> +\t\t\tif (process_request())\n> +\t\t\t\tbreak;\n> +\t}\n> +}\n> diff --git a/serve.h b/serve.h\n> new file mode 100644\n> index 000000000..fe65ba9f4\n> --- /dev/null\n> +++ b/serve.h\n> @@ -0,0 +1,15 @@\n> +#ifndef SERVE_H\n> +#define SERVE_H\n> +\n> +struct argv_array;\n> +extern int has_capability(const struct argv_array *keys, const char *capability,\n> +\t\t\t const char **value);\n> +\n> +struct serve_options {\n> +\tunsigned advertise_capabilities;\n> +\tunsigned stateless_rpc;\n> +};\n> +#define SERVE_OPTIONS_INIT { 0 }\n> +extern void serve(struct serve_options *options);\n> +\n> +#endif /* SERVE_H */\n> diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\n> new file mode 100755\n> index 000000000..b5cc049e5\n> --- /dev/null\n> +++ b/t/t5701-git-serve.sh\n> @@ -0,0 +1,56 @@\n> +#!/bin/sh\n> +\n> +test_description='test git-serve and server commands'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'test capability advertisement' '\n> +\tcat >expect <<-EOF &&\n> +\tversion 2\n> +\tagent=git/$(git version | cut -d\" \" -f3)\n> +\tstateless-rpc\n> +\t0000\n> +\tEOF\n> +\n> +\tgit serve --advertise-capabilities >out &&\n> +\ttest-pkt-line unpack actual &&\n> +\ttest_cmp actual expect\n> +'\n> +\n> +test_expect_success 'stateless-rpc flag does not list capabilities' '\n> +\ttest-pkt-line pack >in <<-EOF &&\n> +\t0000\n> +\tEOF\n> +\tgit serve --stateless-rpc >out +\ttest_must_be_empty out\n> +'\n> +\n> +test_expect_success 'request invalid capability' '\n> +\ttest-pkt-line pack >in <<-EOF &&\n> +\tfoobar\n> +\t0000\n> +\tEOF\n> +\ttest_must_fail git serve --stateless-rpc 2>err +\ttest_i18ngrep \"unknown capability\" err\n> +'\n> +\n> +test_expect_success 'request with no command' '\n> +\ttest-pkt-line pack >in <<-EOF &&\n> +\tagent=git/test\n> +\t0000\n> +\tEOF\n> +\ttest_must_fail git serve --stateless-rpc 2>err +\ttest_i18ngrep \"no command requested\" err\n> +'\n> +\n> +test_expect_success 'request invalid command' '\n> +\ttest-pkt-line pack >in <<-EOF &&\n> +\tcommand=foo\n> +\tagent=git/test\n> +\t0000\n> +\tEOF\n> +\ttest_must_fail git serve --stateless-rpc 2>err +\ttest_i18ngrep \"invalid command\" err\n> +'\n> +\n> +test_done\n\n"},{"id":"337963","messageId":"56044237-4086-faeb-730e-807858a082a8@gmail.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"Re: [PATCH v2 00/27] protocol version 2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-31T16:00:27Z","receivedAt":"2018-01-31T16:00:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Sorry for chiming in with mostly nitpicks so late since sending this \nversion. Mostly, I tried to read it to see if I could understand the \nscope of the patch and how this code worked before. It looks very \npolished, so I the nits were the best I could do.\n\nOn 1/25/2018 6:58 PM, Brandon Williams wrote:\n> Changes in v2:\n> * Added documentation for fetch\n> * changes #defines for state variables to be enums\n> * couple code changes to pkt-line functions and documentation\n> * Added unit tests for the git-serve binary as well as for ls-refs\n\nI'm a fan of more unit-level testing, and I think that will be more \nimportant as we go on with these multiple configuration options.\n\n> Areas for improvement\n> * Push isn't implemented, right now this is ok because if v2 is requested the\n> server can just default to v0. Before this can be merged we may want to\n> change how the client request a new protocol, and not allow for sending\n> \"version=2\" when pushing even though the user has it configured. Or maybe\n> its fine to just have an older client who doesn't understand how to push\n> (and request v2) to die if the server tries to speak v2 at it.\n>\n> Fixing this essentially would just require piping through a bit more\n> information to the function which ultimately runs connect (for both builtins\n> and remote-curl)\n\nDefinitely save push for a later patch. Getting 'fetch' online did \nrequire 'ls-refs' at the same time. Future reviews will be easier when \nadding one command at a time.\n\n>\n> * I want to make sure that the docs are well written before this gets merged\n> so I'm hoping that someone can do a through review on the docs themselves to\n> make sure they are clear.\n\nI made a comment in the docs about the architectural changes. While I \nthink a discussion on that topic would be valuable, I'm not sure that's \nthe point of the document (i.e. documenting what v2 does versus selling \nthe value of the patch). I thought the docs were clear for how the \ncommands work.\n\n> * Right now there is a capability 'stateless-rpc' which essentially makes sure\n> that a server command completes after a single round (this is to make sure\n> http works cleanly). After talking with some folks it may make more sense\n> to just have v2 be stateless in nature so that all commands terminate after\n> a single round trip. This makes things a bit easier if a server wants to\n> have ssh just be a proxy for http.\n>\n> One potential thing would be to flip this so that by default the protocol is\n> stateless and if a server/command has a state-full mode that can be\n> implemented as a capability at a later point. Thoughts?\n\nAt minimum, all commands should be designed with a \"stateless first\" \nphilosophy since a large number of users communicate via HTTP[S] and any \ndecisions that make stateless communication painful should be rejected.\n\n> * Shallow repositories and shallow clones aren't supported yet. I'm working\n> on it and it can be either added to v2 by default if people think it needs\n> to be in there from the start, or we can add it as a capability at a later\n> point.\n\nI'm happy to say the following:\n\n1. Shallow repositories should not be used for servers, since they \ncannot service all requests.\n\n2. Since v2 has easy capability features, I'm happy to leave shallow for \nlater. We will want to verify that a shallow clone command reverts to v1.\n\n\nI fetched bw/protocol-v2 with tip 13c70148, built, set \n'protocol.version=2' in the config, and tested fetches against GitHub \nand VSTS just as a compatibility test. Everything worked just fine.\n\nIs there an easy way to test the existing test suite for clone and fetch \nusing protocol v2 to make sure there are no regressions with \nprotocol.version=2 in the config?\n\nThanks,\n-Stolee\n"},{"id":"337972","messageId":"CAPig+cQvPdMtAWijJaez17m4_fdG6ho_EJW9t=KmnTpocquwgg@mail.gmail.com","threadId":"47524","inReplyTo":"2007efc8-ec31-0267-9b83-392a9d2eaf4a@gmail.com","subject":"Re: [PATCH v2 14/27] connect: request remote refs using v2","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-01-31T20:10:57Z","receivedAt":"2018-01-31T20:11:15Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Jan 31, 2018 at 10:22 AM, Derrick Stolee wrote:\n> On 1/25/2018 6:58 PM, Brandon Williams wrote:\n>> +static int process_ref_v2(const char *line, struct ref ***list)\n>> +{\n>> + int ret = 1;\n>> + int i = 0;\n>\n> nit: you set 'i' here, but first use it in a for loop with blank\n> initializer. Perhaps keep the first assignment closer to the first use?\n\nHmm, I see 'i' being incremented a couple times before the loop...\n\n>> + if (string_list_split(&line_sections, line, ' ', -1) < 2) {\n>> + ret = 0;\n>> + goto out;\n>> + }\n>> +\n>> + if (get_oid_hex(line_sections.items[i++].string, &old_oid)) {\n\nhere...\n\n>> + ret = 0;\n>> + goto out;\n>> + }\n>> +\n>> + ref = alloc_ref(line_sections.items[i++].string);\n\nand here...\n\n>> +\n>> + oidcpy(&ref->old_oid, &old_oid);\n>> + **list = ref;\n>> + *list = &ref->next;\n>> +\n>> + for (; i < line_sections.nr; i++) {\n\nthen it is used in the loop.\n\n>> + const char *arg = line_sections.items[i].string;\n>> + if (skip_prefix(arg, \"symref-target:\", &arg))\n>> + ref->symref = xstrdup(arg);\n"},{"id":"337979","messageId":"514f4a9a-6c88-1dbd-e126-de4dc4c31a8f@gmail.com","threadId":"47524","inReplyTo":"CAPig+cQvPdMtAWijJaez17m4_fdG6ho_EJW9t=KmnTpocquwgg@mail.gmail.com","subject":"Re: [PATCH v2 14/27] connect: request remote refs using v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-31T22:14:33Z","receivedAt":"2018-01-31T22:14:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\n\nOn 1/31/2018 3:10 PM, Eric Sunshine wrote:\n> On Wed, Jan 31, 2018 at 10:22 AM, Derrick Stolee wrote:\n>> On 1/25/2018 6:58 PM, Brandon Williams wrote:\n>>> +static int process_ref_v2(const char *line, struct ref ***list)\n>>> +{\n>>> + int ret = 1;\n>>> + int i = 0;\n>> nit: you set 'i' here, but first use it in a for loop with blank\n>> initializer. Perhaps keep the first assignment closer to the first use?\n> Hmm, I see 'i' being incremented a couple times before the loop...\n>\n>>> + if (string_list_split(&line_sections, line, ' ', -1) < 2) {\n>>> + ret = 0;\n>>> + goto out;\n>>> + }\n>>> +\n>>> + if (get_oid_hex(line_sections.items[i++].string, &old_oid)) {\n> here...\n>\n>>> + ret = 0;\n>>> + goto out;\n>>> + }\n>>> +\n>>> + ref = alloc_ref(line_sections.items[i++].string);\n> and here...\n>\n>>> +\n>>> + oidcpy(&ref->old_oid, &old_oid);\n>>> + **list = ref;\n>>> + *list = &ref->next;\n>>> +\n>>> + for (; i < line_sections.nr; i++) {\n> then it is used in the loop.\n>\n>>> + const char *arg = line_sections.items[i].string;\n>>> + if (skip_prefix(arg, \"symref-target:\", &arg))\n>>> + ref->symref = xstrdup(arg);\n\nThanks! Sorry I missed this.\n\n-Stolee\n"},{"id":"338033","messageId":"20180201175705.GA138727@google.com","threadId":"47524","inReplyTo":"223c0249-24e3-d714-faa5-3c7166b92e99@gmail.com","subject":"Re: [PATCH v2 08/27] connect: discover protocol version outside of get_remote_heads","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-01T17:57:05Z","receivedAt":"2018-02-01T17:57:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/31, Derrick Stolee wrote:\n> On 1/25/2018 6:58 PM, Brandon Williams wrote:\n> > In order to prepare for the addition of protocol_v2 push the protocol\n> > version discovery outside of 'get_remote_heads()'. This will allow for\n> > keeping the logic for processing the reference advertisement for\n> > protocol_v1 and protocol_v0 separate from the logic for protocol_v2.\n> > \n> > Signed-off-by: Brandon Williams \n> > ---\n> > builtin/fetch-pack.c | 16 +++++++++++++++-\n> > builtin/send-pack.c | 17 +++++++++++++++--\n> > connect.c | 27 ++++++++++-----------------\n> > connect.h | 3 +++\n> > remote-curl.c | 20 ++++++++++++++++++--\n> > remote.h | 5 +++--\n> > transport.c | 24 +++++++++++++++++++-----\n> > 7 files changed, 83 insertions(+), 29 deletions(-)\n> > \n> > diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n> > index 366b9d13f..85d4faf76 100644\n> > --- a/builtin/fetch-pack.c\n> > +++ b/builtin/fetch-pack.c\n> > @@ -4,6 +4,7 @@\n> > #include \"remote.h\"\n> > #include \"connect.h\"\n> > #include \"sha1-array.h\"\n> > +#include \"protocol.h\"\n> > static const char fetch_pack_usage[] =\n> > \"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] \"\n> > @@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> > \tstruct fetch_pack_args args;\n> > \tstruct oid_array shallow = OID_ARRAY_INIT;\n> > \tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n> > +\tstruct packet_reader reader;\n> > \tpacket_trace_identity(\"fetch-pack\");\n> > @@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> > \t\tif (!conn)\n> > \t\t\treturn args.diag_url ? 0 : 1;\n> > \t}\n> > -\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n> > +\n> > +\tpacket_reader_init(&reader, fd[0], NULL, 0,\n> > +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> > +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > +\n> > +\tswitch (discover_version(&reader)) {\n> > +\tcase protocol_v1:\n> > +\tcase protocol_v0:\n> > +\t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n> > +\t\tbreak;\n> > +\tcase protocol_unknown_version:\n> > +\t\tBUG(\"unknown protocol version\");\n> \n> Is this really a BUG in the client, or a bug/incompatibility in the server?\n> \n> Perhaps I'm misunderstanding, but it looks like discover_version() will\n> die() on an unknown version (the die() is in\n> protocol.c:determine_protocol_version_client()). So maybe that's why this is\n> a BUG()?\n> \n> If there is something to change here, this BUG() appears three more times.\n\nYes, I have it labeled as a BUG because discover_version can't return an\nunknown protocol version. If the server actually returns an unknown\nprotocol version then it should be handled in\nprotocol.c:determine_protocol_version_client() as you mentioned.\n\n> \n> > +\t}\n> > \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n> > \t\t\t &shallow, pack_lockfile_ptr);\n> > diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n> > index fc4f0bb5f..83cb125a6 100644\n> > --- a/builtin/send-pack.c\n> > +++ b/builtin/send-pack.c\n> > @@ -14,6 +14,7 @@\n> > #include \"sha1-array.h\"\n> > #include \"gpg-interface.h\"\n> > #include \"gettext.h\"\n> > +#include \"protocol.h\"\n> > static const char * const send_pack_usage[] = {\n> > \tN_(\"git send-pack [--all | --mirror] [--dry-run] [--force] \"\n> > @@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n> > \tint progress = -1;\n> > \tint from_stdin = 0;\n> > \tstruct push_cas_option cas = {0};\n> > +\tstruct packet_reader reader;\n> > \tstruct option options[] = {\n> > \t\tOPT__VERBOSITY(&verbose),\n> > @@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n> > \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n> > \t}\n> > -\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n> > -\t\t\t &extra_have, &shallow);\n> > +\tpacket_reader_init(&reader, fd[0], NULL, 0,\n> > +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> > +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > +\n> > +\tswitch (discover_version(&reader)) {\n> > +\tcase protocol_v1:\n> > +\tcase protocol_v0:\n> > +\t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n> > +\t\t\t\t &extra_have, &shallow);\n> > +\t\tbreak;\n> > +\tcase protocol_unknown_version:\n> > +\t\tBUG(\"unknown protocol version\");\n> > +\t}\n> > \ttransport_verify_remote_names(nr_refspecs, refspecs);\n> > diff --git a/connect.c b/connect.c\n> > index 00e90075c..db3c9d24c 100644\n> > --- a/connect.c\n> > +++ b/connect.c\n> > @@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)\n> > \t\t \"and the repository exists.\"));\n> > }\n> > -static enum protocol_version discover_version(struct packet_reader *reader)\n> > +enum protocol_version discover_version(struct packet_reader *reader)\n> > {\n> > \tenum protocol_version version = protocol_unknown_version;\n> > @@ -234,7 +234,7 @@ enum get_remote_heads_state {\n> > /*\n> > * Read all the refs from the other end\n> > */\n> > -struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > +struct ref **get_remote_heads(struct packet_reader *reader,\n> > \t\t\t struct ref **list, unsigned int flags,\n> > \t\t\t struct oid_array *extra_have,\n> > \t\t\t struct oid_array *shallow_points)\n> > @@ -242,24 +242,17 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > \tstruct ref **orig_list = list;\n> > \tint len = 0;\n> > \tenum get_remote_heads_state state = EXPECTING_FIRST_REF;\n> > -\tstruct packet_reader reader;\n> > \tconst char *arg;\n> > -\tpacket_reader_init(&reader, in, src_buf, src_len,\n> > -\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> > -\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > -\n> > -\tdiscover_version(&reader);\n> > -\n> > \t*list = NULL;\n> > \twhile (state != EXPECTING_DONE) {\n> > -\t\tswitch (packet_reader_read(&reader)) {\n> > +\t\tswitch (packet_reader_read(reader)) {\n> > \t\tcase PACKET_READ_EOF:\n> > \t\t\tdie_initial_contact(1);\n> > \t\tcase PACKET_READ_NORMAL:\n> > -\t\t\tlen = reader.pktlen;\n> > -\t\t\tif (len > 4 && skip_prefix(reader.line, \"ERR \", &arg))\n> > +\t\t\tlen = reader->pktlen;\n> > +\t\t\tif (len > 4 && skip_prefix(reader->line, \"ERR \", &arg))\n> > \t\t\t\tdie(\"remote error: %s\", arg);\n> > \t\t\tbreak;\n> > \t\tcase PACKET_READ_FLUSH:\n> > @@ -271,22 +264,22 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > \t\tswitch (state) {\n> > \t\tcase EXPECTING_FIRST_REF:\n> > -\t\t\tprocess_capabilities(reader.line, &len);\n> > -\t\t\tif (process_dummy_ref(reader.line)) {\n> > +\t\t\tprocess_capabilities(reader->line, &len);\n> > +\t\t\tif (process_dummy_ref(reader->line)) {\n> > \t\t\t\tstate = EXPECTING_SHALLOW;\n> > \t\t\t\tbreak;\n> > \t\t\t}\n> > \t\t\tstate = EXPECTING_REF;\n> > \t\t\t/* fallthrough */\n> > \t\tcase EXPECTING_REF:\n> > -\t\t\tif (process_ref(reader.line, len, &list, flags, extra_have))\n> > +\t\t\tif (process_ref(reader->line, len, &list, flags, extra_have))\n> > \t\t\t\tbreak;\n> > \t\t\tstate = EXPECTING_SHALLOW;\n> > \t\t\t/* fallthrough */\n> > \t\tcase EXPECTING_SHALLOW:\n> > -\t\t\tif (process_shallow(reader.line, len, shallow_points))\n> > +\t\t\tif (process_shallow(reader->line, len, shallow_points))\n> > \t\t\t\tbreak;\n> > -\t\t\tdie(\"protocol error: unexpected '%s'\", reader.line);\n> > +\t\t\tdie(\"protocol error: unexpected '%s'\", reader->line);\n> > \t\tcase EXPECTING_DONE:\n> > \t\t\tbreak;\n> > \t\t}\n> > diff --git a/connect.h b/connect.h\n> > index 01f14cdf3..cdb8979dc 100644\n> > --- a/connect.h\n> > +++ b/connect.h\n> > @@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);\n> > extern const char *server_feature_value(const char *feature, int *len_ret);\n> > extern int url_is_local_not_ssh(const char *url);\n> > +struct packet_reader;\n> > +extern enum protocol_version discover_version(struct packet_reader *reader);\n> > +\n> > #endif\n> > diff --git a/remote-curl.c b/remote-curl.c\n> > index 0053b0954..9f6d07683 100644\n> > --- a/remote-curl.c\n> > +++ b/remote-curl.c\n> > @@ -1,6 +1,7 @@\n> > #include \"cache.h\"\n> > #include \"config.h\"\n> > #include \"remote.h\"\n> > +#include \"connect.h\"\n> > #include \"strbuf.h\"\n> > #include \"walker.h\"\n> > #include \"http.h\"\n> > @@ -13,6 +14,7 @@\n> > #include \"credential.h\"\n> > #include \"sha1-array.h\"\n> > #include \"send-pack.h\"\n> > +#include \"protocol.h\"\n> > static struct remote *remote;\n> > /* always ends with a trailing slash */\n> > @@ -176,8 +178,22 @@ static struct discovery *last_discovery;\n> > static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n> > {\n> > \tstruct ref *list = NULL;\n> > -\tget_remote_heads(-1, heads->buf, heads->len, &list,\n> > -\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n> > +\tstruct packet_reader reader;\n> > +\n> > +\tpacket_reader_init(&reader, -1, heads->buf, heads->len,\n> > +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> > +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > +\n> > +\tswitch (discover_version(&reader)) {\n> > +\tcase protocol_v1:\n> > +\tcase protocol_v0:\n> > +\t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n> > +\t\t\t\t NULL, &heads->shallow);\n> > +\t\tbreak;\n> > +\tcase protocol_unknown_version:\n> > +\t\tBUG(\"unknown protocol version\");\n> > +\t}\n> > +\n> > \treturn list;\n> > }\n> > diff --git a/remote.h b/remote.h\n> > index 1f6611be2..2016461df 100644\n> > --- a/remote.h\n> > +++ b/remote.h\n> > @@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n> > void free_refs(struct ref *ref);\n> > struct oid_array;\n> > -extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > +struct packet_reader;\n> > +extern struct ref **get_remote_heads(struct packet_reader *reader,\n> > \t\t\t\t struct ref **list, unsigned int flags,\n> > \t\t\t\t struct oid_array *extra_have,\n> > -\t\t\t\t struct oid_array *shallow);\n> > +\t\t\t\t struct oid_array *shallow_points);\n> > int resolve_remote_symref(struct ref *ref, struct ref *list);\n> > int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\n> > diff --git a/transport.c b/transport.c\n> > index 8e8779096..63c3dbab9 100644\n> > --- a/transport.c\n> > +++ b/transport.c\n> > @@ -18,6 +18,7 @@\n> > #include \"sha1-array.h\"\n> > #include \"sigchain.h\"\n> > #include \"transport-internal.h\"\n> > +#include \"protocol.h\"\n> > static void set_upstreams(struct transport *transport, struct ref *refs,\n> > \tint pretend)\n> > @@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)\n> > static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n> > {\n> > \tstruct git_transport_data *data = transport->data;\n> > -\tstruct ref *refs;\n> > +\tstruct ref *refs = NULL;\n> > +\tstruct packet_reader reader;\n> > \tconnect_setup(transport, for_push);\n> > -\tget_remote_heads(data->fd[0], NULL, 0, &refs,\n> > -\t\t\t for_push ? REF_NORMAL : 0,\n> > -\t\t\t &data->extra_have,\n> > -\t\t\t &data->shallow);\n> > +\n> > +\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n> > +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> > +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > +\n> > +\tswitch (discover_version(&reader)) {\n> > +\tcase protocol_v1:\n> > +\tcase protocol_v0:\n> > +\t\tget_remote_heads(&reader, &refs,\n> > +\t\t\t\t for_push ? REF_NORMAL : 0,\n> > +\t\t\t\t &data->extra_have,\n> > +\t\t\t\t &data->shallow);\n> > +\t\tbreak;\n> > +\tcase protocol_unknown_version:\n> > +\t\tBUG(\"unknown protocol version\");\n> > +\t}\n> > \tdata->got_remote_heads = 1;\n> > \treturn refs;\n> \n\n-- \nBrandon Williams\n"},{"id":"338034","messageId":"d9a15c0a-35f3-8e9c-ddf2-34420ac7555b@jeffhostetler.com","threadId":"47524","inReplyTo":"20180103001828.205012-12-bmwill@google.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-02-01T18:48:59Z","receivedAt":"2018-02-01T18:49:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 1/2/2018 7:18 PM, Brandon Williams wrote:\n> Introduce git-serve, the base server for protocol version 2.\n> \n> Protocol version 2 is intended to be a replacement for Git's current\n> wire protocol. The intention is that it will be a simpler, less\n> wasteful protocol which can evolve over time.\n> \n> Protocol version 2 improves upon version 1 by eliminating the initial\n> ref advertisement. In its place a server will export a list of\n> capabilities and commands which it supports in a capability\n> advertisement. A client can then request that a particular command be\n> executed by providing a number of capabilities and command specific\n> parameters. At the completion of a command, a client can request that\n> another command be executed or can terminate the connection by sending a\n> flush packet.\n> \n> Signed-off-by: Brandon Williams \n> ---\n> .gitignore | 1 +\n> Documentation/technical/protocol-v2.txt | 91 ++++++++++++\n> Makefile | 2 +\n> builtin.h | 1 +\n> builtin/serve.c | 30 ++++\n> git.c | 1 +\n> serve.c | 239 ++++++++++++++++++++++++++++++++\n> serve.h | 15 ++\n> 8 files changed, 380 insertions(+)\n> create mode 100644 Documentation/technical/protocol-v2.txt\n> create mode 100644 builtin/serve.c\n> create mode 100644 serve.c\n> create mode 100644 serve.h\n> \n> diff --git a/.gitignore b/.gitignore\n> index 833ef3b0b..2d0450c26 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -140,6 +140,7 @@\n> /git-rm\n> /git-send-email\n> /git-send-pack\n> +/git-serve\n> /git-sh-i18n\n> /git-sh-i18n--envsubst\n> /git-sh-setup\n> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> new file mode 100644\n> index 000000000..b87ba3816\n> --- /dev/null\n> +++ b/Documentation/technical/protocol-v2.txt\n> @@ -0,0 +1,91 @@\n> + Git Wire Protocol, Version 2\n> +==============================\n> +\n> +This document presents a specification for a version 2 of Git's wire\n> +protocol. Protocol v2 will improve upon v1 in the following ways:\n> +\n> + * Instead of multiple service names, multiple commands will be\n> + supported by a single service.\n> + * Easily extendable as capabilities are moved into their own section\n> + of the protocol, no longer being hidden behind a NUL byte and\n> + limited by the size of a pkt-line (as there will be a single\n> + capability per pkt-line).\n> + * Separate out other information hidden behind NUL bytes (e.g. agent\n> + string as a capability and symrefs can be requested using 'ls-refs')\n> + * Reference advertisement will be omitted unless explicitly requested\n> + * ls-refs command to explicitly request some refs\n> +\n> + Detailed Design\n> +=================\n> +\n> +A client can request to speak protocol v2 by sending `version=2` in the\n> +side-channel `GIT_PROTOCOL` in the initial request to the server.\n> +\n> +In protocol v2 communication is command oriented. When first contacting a\n> +server a list of capabilities will advertised. Some of these capabilities\n> +will be commands which a client can request be executed. Once a command\n> +has completed, a client can reuse the connection and request that other\n> +commands be executed.\n> +\n> + Special Packets\n> +-----------------\n> +\n> +In protocol v2 these special packets will have the following semantics:\n> +\n> + * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n> + * '0001' Delimiter Packet (delim-pkt) - separates sections of a message\n\nPreviously, a 0001 pkt-line meant that there was 1 byte of data\nfollowing, right? Does this change that and/or prevent 1 byte\npackets? (Not sure if it is likely, but the odd-tail of a packfile\nmight get sent in a 0001 line, right?) Or is it that 0001 is only\nspecial during the V2 negotiation stuff, but not during the packfile\ntransmission?\n\n(I'm not against having this delimiter -- I think it is useful, but\njust curious if will cause problems elsewhere.)\n\nShould we also consider increasing the pkt-line limit to 5 hex-digits\nwhile we're at it ? That would let us have 1MB buffers if that would\nhelp with large packfiles. Granted, we're throttled by the network,\nso it might not matter. Would it be interesting to have a 5 digit\nprefix with parts of the high bits of first digit being flags ?\nOr is this too radical of a change?\n\n\n> +\n> + Capability Advertisement\n> +--------------------------\n> +\n> +A server which decides to communicate (based on a request from a client)\n> +using protocol version 2, notifies the client by sending a version string\n> +in its initial response followed by an advertisement of its capabilities.\n> +Each capability is a key with an optional value. Clients must ignore all\n> +unknown keys. Semantics of unknown values are left to the definition of\n> +each key. Some capabilities will describe commands which can be requested\n> +to be executed by the client.\n> +\n> + capability-advertisement = protocol-version\n> +\t\t\t capability-list\n> +\t\t\t flush-pkt\n> +\n> + protocol-version = PKT-LINE(\"version 2\" LF)\n> + capability-list = *capability\n> + capability = PKT-LINE(key[=value] LF)\n> +\n> + key = 1*CHAR\n> + value = 1*CHAR\n> + CHAR = 1*(ALPHA / DIGIT / \"-\" / \"_\")\n> +\n> +A client then responds to select the command it wants with any particular\n> +capabilities or arguments. There is then an optional section where the\n> +client can provide any command specific parameters or queries.\n> +\n> + command-request = command\n> +\t\t capability-list\n> +\t\t (command-args)\n> +\t\t flush-pkt\n> + command = PKT-LINE(\"command=\" key LF)\n> + command-args = delim-pkt\n> +\t\t *arg\n> + arg = 1*CHAR\n> +\n> +The server will then check to ensure that the client's request is\n> +comprised of a valid command as well as valid capabilities which were\n> +advertised. If the request is valid the server will then execute the\n> +command.\n> +\n> +A particular command can last for as many rounds as are required to\n> +complete the service (multiple for negotiation during fetch or no\n> +additional trips in the case of ls-refs).\n> +\n> +When finished a client should send an empty request of just a flush-pkt to\n> +terminate the connection.\n> +\n> + Commands in v2\n> +~~~~~~~~~~~~~~~~\n> +\n> +Commands are the core actions that a client wants to perform (fetch, push,\n> +etc). Each command will be provided with a list capabilities and\n> +arguments as requested by a client.\n\n"},{"id":"338036","messageId":"CAGZ79kZpD6m=5YP0dNuM70Yce=PyxLO6qr6aVpHKa_S9iyACww@mail.gmail.com","threadId":"47524","inReplyTo":"d9a15c0a-35f3-8e9c-ddf2-34420ac7555b@jeffhostetler.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-01T18:57:38Z","receivedAt":"2018-02-01T18:57:44Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Feb 1, 2018 at 10:48 AM, Jeff Hostetler wrote:\n>\n>\n> On 1/2/2018 7:18 PM, Brandon Williams wrote:\n>>\n>> Introduce git-serve, the base server for protocol version 2.\n>>\n>> Protocol version 2 is intended to be a replacement for Git's current\n>> wire protocol. The intention is that it will be a simpler, less\n>> wasteful protocol which can evolve over time.\n>>\n>> Protocol version 2 improves upon version 1 by eliminating the initial\n>> ref advertisement. In its place a server will export a list of\n>> capabilities and commands which it supports in a capability\n>> advertisement. A client can then request that a particular command be\n>> executed by providing a number of capabilities and command specific\n>> parameters. At the completion of a command, a client can request that\n>> another command be executed or can terminate the connection by sending a\n>> flush packet.\n>>\n>> Signed-off-by: Brandon Williams \n>> ---\n>> .gitignore | 1 +\n>> Documentation/technical/protocol-v2.txt | 91 ++++++++++++\n>> Makefile | 2 +\n>> builtin.h | 1 +\n>> builtin/serve.c | 30 ++++\n>> git.c | 1 +\n>> serve.c | 239\n>> ++++++++++++++++++++++++++++++++\n>> serve.h | 15 ++\n>> 8 files changed, 380 insertions(+)\n>> create mode 100644 Documentation/technical/protocol-v2.txt\n>> create mode 100644 builtin/serve.c\n>> create mode 100644 serve.c\n>> create mode 100644 serve.h\n>>\n>> diff --git a/.gitignore b/.gitignore\n>> index 833ef3b0b..2d0450c26 100644\n>> --- a/.gitignore\n>> +++ b/.gitignore\n>> @@ -140,6 +140,7 @@\n>> /git-rm\n>> /git-send-email\n>> /git-send-pack\n>> +/git-serve\n>> /git-sh-i18n\n>> /git-sh-i18n--envsubst\n>> /git-sh-setup\n>> diff --git a/Documentation/technical/protocol-v2.txt\n>> b/Documentation/technical/protocol-v2.txt\n>> new file mode 100644\n>> index 000000000..b87ba3816\n>> --- /dev/null\n>> +++ b/Documentation/technical/protocol-v2.txt\n>> @@ -0,0 +1,91 @@\n>> + Git Wire Protocol, Version 2\n>> +==============================\n>> +\n>> +This document presents a specification for a version 2 of Git's wire\n>> +protocol. Protocol v2 will improve upon v1 in the following ways:\n>> +\n>> + * Instead of multiple service names, multiple commands will be\n>> + supported by a single service.\n>> + * Easily extendable as capabilities are moved into their own section\n>> + of the protocol, no longer being hidden behind a NUL byte and\n>> + limited by the size of a pkt-line (as there will be a single\n>> + capability per pkt-line).\n>> + * Separate out other information hidden behind NUL bytes (e.g. agent\n>> + string as a capability and symrefs can be requested using 'ls-refs')\n>> + * Reference advertisement will be omitted unless explicitly requested\n>> + * ls-refs command to explicitly request some refs\n>> +\n>> + Detailed Design\n>> +=================\n>> +\n>> +A client can request to speak protocol v2 by sending `version=2` in the\n>> +side-channel `GIT_PROTOCOL` in the initial request to the server.\n>> +\n>> +In protocol v2 communication is command oriented. When first contacting\n>> a\n>> +server a list of capabilities will advertised. Some of these\n>> capabilities\n>> +will be commands which a client can request be executed. Once a command\n>> +has completed, a client can reuse the connection and request that other\n>> +commands be executed.\n>> +\n>> + Special Packets\n>> +-----------------\n>> +\n>> +In protocol v2 these special packets will have the following semantics:\n>> +\n>> + * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n>> + * '0001' Delimiter Packet (delim-pkt) - separates sections of a message\n>\n>\n> Previously, a 0001 pkt-line meant that there was 1 byte of data\n> following, right?\n\nNo, the length was including the length field, so 0005 would indicate that\nthere is one byte following, (+4 bytes of \"0005\" included)\n\n> Does this change that and/or prevent 1 byte\n> packets? (Not sure if it is likely, but the odd-tail of a packfile\n> might get sent in a 0001 line, right?) Or is it that 0001 is only\n> special during the V2 negotiation stuff, but not during the packfile\n> transmission?\n\n0001 is invalid in the current protocol v0.\n\n>\n> (I'm not against having this delimiter -- I think it is useful, but\n> just curious if will cause problems elsewhere.)\n>\n> Should we also consider increasing the pkt-line limit to 5 hex-digits\n> while we're at it ? That would let us have 1MB buffers if that would\n> help with large packfiles.\n\nAFAICT there is a static allocation of one pkt-line (of maximum size),\nsuch that the code can read in a full packet and then process it.\nIf we'd increase the packet size we'd need the static buffer to be 1MB,\nwhich sounds good for my developer machine. But I suspect it may be\ntoo much for people using git on embedded devices?\n\npack files larger than 64k are put into multiple pkt-lines, which is\nnot a big deal, as the overhead of 4bytes per 64k is negligible.\n(also there is progress information in the side channel, which\nwould come in as a special packet in between real packets,\nsuch that every 64k transmitted you can update your progress\nmeter; Not sure I feel strongly on fewer progress updates)\n\n> Granted, we're throttled by the network,\n> so it might not matter. Would it be interesting to have a 5 digit\n> prefix with parts of the high bits of first digit being flags ?\n> Or is this too radical of a change?\n\nWhat would the flags be for?\n\nAs an alternative we could put the channel number in one byte,\nsuch that we can have a side channel not just while streaming the\npack but all the time. (Again, not sure if that buys a lot for us)\n"},{"id":"338037","messageId":"af1e00f7-b91d-4c1e-bc0e-164371feeadc@jeffhostetler.com","threadId":"47524","inReplyTo":"CAGZ79kZpD6m=5YP0dNuM70Yce=PyxLO6qr6aVpHKa_S9iyACww@mail.gmail.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-02-01T19:09:45Z","receivedAt":"2018-02-01T19:09:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/1/2018 1:57 PM, Stefan Beller wrote:\n> On Thu, Feb 1, 2018 at 10:48 AM, Jeff Hostetler wrote:\n>>\n>>\n>> On 1/2/2018 7:18 PM, Brandon Williams wrote:\n>>>\n>>> Introduce git-serve, the base server for protocol version 2.\n[...]\n>>> + Special Packets\n>>> +-----------------\n>>> +\n>>> +In protocol v2 these special packets will have the following semantics:\n>>> +\n>>> + * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n>>> + * '0001' Delimiter Packet (delim-pkt) - separates sections of a message\n>>\n>>\n>> Previously, a 0001 pkt-line meant that there was 1 byte of data\n>> following, right?\n> \n> No, the length was including the length field, so 0005 would indicate that\n> there is one byte following, (+4 bytes of \"0005\" included)\n\nd'oh. right. thanks!\n\n>> Should we also consider increasing the pkt-line limit to 5 hex-digits\n>> while we're at it ? That would let us have 1MB buffers if that would\n>> help with large packfiles.\n> \n> AFAICT there is a static allocation of one pkt-line (of maximum size),\n> such that the code can read in a full packet and then process it.\n> If we'd increase the packet size we'd need the static buffer to be 1MB,\n> which sounds good for my developer machine. But I suspect it may be\n> too much for people using git on embedded devices?\n\nI got burned by that static buffer once upon a time when I wanted\nto have 2 streams going at the same time. Hopefully, we can move\nthat into the new reader structure at some point (if it isn't already).\n\n> \n> pack files larger than 64k are put into multiple pkt-lines, which is\n> not a big deal, as the overhead of 4bytes per 64k is negligible.\n> (also there is progress information in the side channel, which\n> would come in as a special packet in between real packets,\n> such that every 64k transmitted you can update your progress\n> meter; Not sure I feel strongly on fewer progress updates)\n> \n>> Granted, we're throttled by the network,\n>> so it might not matter. Would it be interesting to have a 5 digit\n>> prefix with parts of the high bits of first digit being flags ?\n>> Or is this too radical of a change?\n> \n> What would the flags be for?\n> \n> As an alternative we could put the channel number in one byte,\n> such that we can have a side channel not just while streaming the\n> pack but all the time. (Again, not sure if that buys a lot for us)\n> \n\nDelimiters like the 0001 and the side channel are a couple of\nideas, but I was just thinking out loud. And right, I'm not sure\nit gets us much right now.\n\nJeff\n"},{"id":"338038","messageId":"e2fb1a05-3cbd-5226-7796-c5b89c9d3955@jeffhostetler.com","threadId":"47524","inReplyTo":"20180103001828.205012-13-bmwill@google.com","subject":"Re: [PATCH 12/26] ls-refs: introduce ls-refs server command","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-02-01T19:16:22Z","receivedAt":"2018-02-01T19:16:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 1/2/2018 7:18 PM, Brandon Williams wrote:\n> Introduce the ls-refs server command. In protocol v2, the ls-refs\n> command is used to request the ref advertisement from the server. Since\n> it is a command which can be requested (as opposed to mandatory in v1),\n> a client can sent a number of parameters in its request to limit the ref\n> advertisement based on provided ref-patterns.\n> \n> Signed-off-by: Brandon Williams \n> ---\n> Documentation/technical/protocol-v2.txt | 26 +++++++++\n> Makefile | 1 +\n> ls-refs.c | 97 +++++++++++++++++++++++++++++++++\n> ls-refs.h | 9 +++\n> serve.c | 2 +\n> 5 files changed, 135 insertions(+)\n> create mode 100644 ls-refs.c\n> create mode 100644 ls-refs.h\n> \n> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> index b87ba3816..5f4d0e719 100644\n> --- a/Documentation/technical/protocol-v2.txt\n> +++ b/Documentation/technical/protocol-v2.txt\n> @@ -89,3 +89,29 @@ terminate the connection.\n> Commands are the core actions that a client wants to perform (fetch, push,\n> etc). Each command will be provided with a list capabilities and\n> arguments as requested by a client.\n> +\n> + Ls-refs\n> +---------\n> +\n> +Ls-refs is the command used to request a reference advertisement in v2.\n> +Unlike the current reference advertisement, ls-refs takes in parameters\n> +which can be used to limit the refs sent from the server.\n> +\n> +Ls-ref takes in the following parameters wraped in packet-lines:\n> +\n> + symrefs: In addition to the object pointed by it, show the underlying\n> +\t ref pointed by it when showing a symbolic ref.\n> + peel: Show peeled tags.\n> + ref-pattern : When specified, only references matching the\n> +\t\t\t given patterns are displayed.\n> +\n> +The output of ls-refs is as follows:\n> +\n> + output = *ref\n> +\t flush-pkt\n> + ref = PKT-LINE((tip | peeled) LF)\n> + tip = obj-id SP refname (SP symref-target)\n> + peeled = obj-id SP refname \"^{}\"\n> +\n> + symref = PKT-LINE(\"symref\" SP symbolic-ref SP resolved-ref LF)\n> + shallow = PKT-LINE(\"shallow\" SP obj-id LF)\n\nDo you want to talk about ordering requirements on this?\nI think packed-refs has one, but I'm not sure it matters here\nwhere the client or server sorts it.\n\nAre there any provisions for compressing the renames, like in the\nreftable spec or in index-v4 ?\n\nIt doesn't need to be in the initial version. Just asking. We could\nalways add a \"ls-refs-2\" command that builds upon this.\n\nThanks,\nJeff\n"},{"id":"338041","messageId":"02c14b43-34b0-44c7-725c-1d0c1ead28e5@jeffhostetler.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"Re: [PATCH v2 00/27] protocol version 2","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-02-01T19:40:07Z","receivedAt":"2018-02-01T19:40:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 1/25/2018 6:58 PM, Brandon Williams wrote:\n> Changes in v2:\n> * Added documentation for fetch\n> * changes #defines for state variables to be enums\n> * couple code changes to pkt-line functions and documentation\n> * Added unit tests for the git-serve binary as well as for ls-refs\n[...]\n\n\nThis looks really nice. I'm eager to get this in so we can do some\nadditional commands to help make partial clone more efficient.\n\nThanks,\nJeff\n\n"},{"id":"338042","messageId":"005801d39b95$2bbc7600$83356200$@nexbridge.com","threadId":"47524","inReplyTo":"CAGZ79kZpD6m=5YP0dNuM70Yce=PyxLO6qr6aVpHKa_S9iyACww@mail.gmail.com","subject":"RE: [PATCH 11/26] serve: introduce git-serve","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-02-01T19:45:05Z","receivedAt":"2018-02-01T19:45:19Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On February 1, 2018 1:58 PM, Stefan Beller wrote:\n> On Thu, Feb 1, 2018 at 10:48 AM, Jeff Hostetler \n> wrote:\n> >\n> >\n> > On 1/2/2018 7:18 PM, Brandon Williams wrote:\n> >>\n> >> Introduce git-serve, the base server for protocol version 2.\n> >>\n> >> Protocol version 2 is intended to be a replacement for Git's current\n> >> wire protocol. The intention is that it will be a simpler, less\n> >> wasteful protocol which can evolve over time.\n> >>\n> >> Protocol version 2 improves upon version 1 by eliminating the initial\n> >> ref advertisement. In its place a server will export a list of\n> >> capabilities and commands which it supports in a capability\n> >> advertisement. A client can then request that a particular command\n> >> be executed by providing a number of capabilities and command\n> >> specific parameters. At the completion of a command, a client can\n> >> request that another command be executed or can terminate the\n> >> connection by sending a flush packet.\n> >>\n> >> Signed-off-by: Brandon Williams \n> >> ---\n> >> .gitignore | 1 +\n> >> Documentation/technical/protocol-v2.txt | 91 ++++++++++++\n> >> Makefile | 2 +\n> >> builtin.h | 1 +\n> >> builtin/serve.c | 30 ++++\n> >> git.c | 1 +\n> >> serve.c | 239\n> >> ++++++++++++++++++++++++++++++++\n> >> serve.h | 15 ++\n> >> 8 files changed, 380 insertions(+)\n> >> create mode 100644 Documentation/technical/protocol-v2.txt\n> >> create mode 100644 builtin/serve.c\n> >> create mode 100644 serve.c\n> >> create mode 100644 serve.h\n> >>\n> >> diff --git a/.gitignore b/.gitignore\n> >> index 833ef3b0b..2d0450c26 100644\n> >> --- a/.gitignore\n> >> +++ b/.gitignore\n> >> @@ -140,6 +140,7 @@\n> >> /git-rm\n> >> /git-send-email\n> >> /git-send-pack\n> >> +/git-serve\n> >> /git-sh-i18n\n> >> /git-sh-i18n--envsubst\n> >> /git-sh-setup\n> >> diff --git a/Documentation/technical/protocol-v2.txt\n> >> b/Documentation/technical/protocol-v2.txt\n> >> new file mode 100644\n> >> index 000000000..b87ba3816\n> >> --- /dev/null\n> >> +++ b/Documentation/technical/protocol-v2.txt\n> >> @@ -0,0 +1,91 @@\n> >> + Git Wire Protocol, Version 2\n> >> +==============================\n> >> +\n> >> +This document presents a specification for a version 2 of Git's wire\n> >> +protocol. Protocol v2 will improve upon v1 in the following ways:\n> >> +\n> >> + * Instead of multiple service names, multiple commands will be\n> >> + supported by a single service.\n> >> + * Easily extendable as capabilities are moved into their own section\n> >> + of the protocol, no longer being hidden behind a NUL byte and\n> >> + limited by the size of a pkt-line (as there will be a single\n> >> + capability per pkt-line).\n> >> + * Separate out other information hidden behind NUL bytes (e.g. agent\n> >> + string as a capability and symrefs can be requested using\n> >> + 'ls-refs')\n> >> + * Reference advertisement will be omitted unless explicitly\n> >> + requested\n> >> + * ls-refs command to explicitly request some refs\n> >> +\n> >> + Detailed Design\n> >> +=================\n> >> +\n> >> +A client can request to speak protocol v2 by sending `version=2` in\n> >> +the side-channel `GIT_PROTOCOL` in the initial request to the server.\n> >> +\n> >> +In protocol v2 communication is command oriented. When first\n> >> +contacting\n> >> a\n> >> +server a list of capabilities will advertised. Some of these\n> >> capabilities\n> >> +will be commands which a client can request be executed. Once a\n> >> +command has completed, a client can reuse the connection and request\n> >> +that other commands be executed.\n> >> +\n> >> + Special Packets\n> >> +-----------------\n> >> +\n> >> +In protocol v2 these special packets will have the following semantics:\n> >> +\n> >> + * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n> >> + * '0001' Delimiter Packet (delim-pkt) - separates sections of a\n> >> + message\n> >\n> >\n> > Previously, a 0001 pkt-line meant that there was 1 byte of data\n> > following, right?\n> \n> No, the length was including the length field, so 0005 would indicate that\n> there is one byte following, (+4 bytes of \"0005\" included)\n> \n> > Does this change that and/or prevent 1 byte packets? (Not sure if it\n> > is likely, but the odd-tail of a packfile might get sent in a 0001\n> > line, right?) Or is it that 0001 is only special during the V2\n> > negotiation stuff, but not during the packfile transmission?\n> \n> 0001 is invalid in the current protocol v0.\n> \n> >\n> > (I'm not against having this delimiter -- I think it is useful, but\n> > just curious if will cause problems elsewhere.)\n> >\n> > Should we also consider increasing the pkt-line limit to 5 hex-digits\n> > while we're at it ? That would let us have 1MB buffers if that would\n> > help with large packfiles.\n> \n> AFAICT there is a static allocation of one pkt-line (of maximum size), such\n> that the code can read in a full packet and then process it.\n> If we'd increase the packet size we'd need the static buffer to be 1MB, which\n> sounds good for my developer machine. But I suspect it may be too much for\n> people using git on embedded devices?\n> \n> pack files larger than 64k are put into multiple pkt-lines, which is not a big\n> deal, as the overhead of 4bytes per 64k is negligible.\n> (also there is progress information in the side channel, which would come in\n> as a special packet in between real packets, such that every 64k transmitted\n> you can update your progress meter; Not sure I feel strongly on fewer\n> progress updates)\n\nCan I request, selfishly from my own platform's (NonStop) performance heartache, that we don't require 1Mb? We're not embedded on this platform, but there is an optimized message system packet size down at 50Kb that I would like to stay under. Although above that is no problem, there is a significant cost incurred above that size point. And please make sure xread/xwrite are used in any event.\n\n> > Granted, we're throttled by the network, so it might not matter.\n> > Would it be interesting to have a 5 digit prefix with parts of the\n> > high bits of first digit being flags ?\n> > Or is this too radical of a change?\n> \n> What would the flags be for?\n> \n> As an alternative we could put the channel number in one byte, such that we\n> can have a side channel not just while streaming the pack but all the time.\n> (Again, not sure if that buys a lot for us)\n\nCheers,\nRandall\n\n-- Brief whoami:\n NonStop developer since approximately 211288444200000000\n UNIX developer since approximately 421664400\n-- In my real life, I talk too much.\n\n\n\n"},{"id":"338043","messageId":"20180201200559.GB138727@google.com","threadId":"47524","inReplyTo":"af1e00f7-b91d-4c1e-bc0e-164371feeadc@jeffhostetler.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-01T20:05:59Z","receivedAt":"2018-02-01T20:06:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/01, Jeff Hostetler wrote:\n> \n> \n> On 2/1/2018 1:57 PM, Stefan Beller wrote:\n> > On Thu, Feb 1, 2018 at 10:48 AM, Jeff Hostetler wrote:\n> > > \n> > > \n> > > On 1/2/2018 7:18 PM, Brandon Williams wrote:\n> > > > \n> > > > Introduce git-serve, the base server for protocol version 2.\n> [...]\n> > > > + Special Packets\n> > > > +-----------------\n> > > > +\n> > > > +In protocol v2 these special packets will have the following semantics:\n> > > > +\n> > > > + * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n> > > > + * '0001' Delimiter Packet (delim-pkt) - separates sections of a message\n> > > \n> > > \n> > > Previously, a 0001 pkt-line meant that there was 1 byte of data\n> > > following, right?\n> > \n> > No, the length was including the length field, so 0005 would indicate that\n> > there is one byte following, (+4 bytes of \"0005\" included)\n> \n> d'oh. right. thanks!\n> \n> > > Should we also consider increasing the pkt-line limit to 5 hex-digits\n> > > while we're at it ? That would let us have 1MB buffers if that would\n> > > help with large packfiles.\n> > \n> > AFAICT there is a static allocation of one pkt-line (of maximum size),\n> > such that the code can read in a full packet and then process it.\n> > If we'd increase the packet size we'd need the static buffer to be 1MB,\n> > which sounds good for my developer machine. But I suspect it may be\n> > too much for people using git on embedded devices?\n> \n> I got burned by that static buffer once upon a time when I wanted\n> to have 2 streams going at the same time. Hopefully, we can move\n> that into the new reader structure at some point (if it isn't already).\n\nYeah the reader struct could easily be extended to take in the\nbuffer to read the data into. Because I'm not trying to do any of that\natm I decided to have it default to using the static buffer, but it\nwould be as simple as changing the reader->buffer variable to use a\ndifferent buffer.\n\n> \n> > \n> > pack files larger than 64k are put into multiple pkt-lines, which is\n> > not a big deal, as the overhead of 4bytes per 64k is negligible.\n> > (also there is progress information in the side channel, which\n> > would come in as a special packet in between real packets,\n> > such that every 64k transmitted you can update your progress\n> > meter; Not sure I feel strongly on fewer progress updates)\n> > \n> > > Granted, we're throttled by the network,\n> > > so it might not matter. Would it be interesting to have a 5 digit\n> > > prefix with parts of the high bits of first digit being flags ?\n> > > Or is this too radical of a change?\n> > \n> > What would the flags be for?\n> > \n> > As an alternative we could put the channel number in one byte,\n> > such that we can have a side channel not just while streaming the\n> > pack but all the time. (Again, not sure if that buys a lot for us)\n> > \n> \n> Delimiters like the 0001 and the side channel are a couple of\n> ideas, but I was just thinking out loud. And right, I'm not sure\n> it gets us much right now.\n> \n> Jeff\n\n-- \nBrandon Williams\n"},{"id":"338044","messageId":"20180201200826.GC138727@google.com","threadId":"47524","inReplyTo":"005801d39b95$2bbc7600$83356200$@nexbridge.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"'Brandon Williams'","fromEmail":"bmwill@google.com","sentAt":"2018-02-01T20:08:26Z","receivedAt":"2018-02-01T20:08:34Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/01, Randall S. Becker wrote:\n> On February 1, 2018 1:58 PM, Stefan Beller wrote:\n> > On Thu, Feb 1, 2018 at 10:48 AM, Jeff Hostetler \n> > wrote:\n> > >\n> > >\n> > > On 1/2/2018 7:18 PM, Brandon Williams wrote:\n> > >>\n> > >> Introduce git-serve, the base server for protocol version 2.\n> > >>\n> > >> Protocol version 2 is intended to be a replacement for Git's current\n> > >> wire protocol. The intention is that it will be a simpler, less\n> > >> wasteful protocol which can evolve over time.\n> > >>\n> > >> Protocol version 2 improves upon version 1 by eliminating the initial\n> > >> ref advertisement. In its place a server will export a list of\n> > >> capabilities and commands which it supports in a capability\n> > >> advertisement. A client can then request that a particular command\n> > >> be executed by providing a number of capabilities and command\n> > >> specific parameters. At the completion of a command, a client can\n> > >> request that another command be executed or can terminate the\n> > >> connection by sending a flush packet.\n> > >>\n> > >> Signed-off-by: Brandon Williams \n> > >> ---\n> > >> .gitignore | 1 +\n> > >> Documentation/technical/protocol-v2.txt | 91 ++++++++++++\n> > >> Makefile | 2 +\n> > >> builtin.h | 1 +\n> > >> builtin/serve.c | 30 ++++\n> > >> git.c | 1 +\n> > >> serve.c | 239\n> > >> ++++++++++++++++++++++++++++++++\n> > >> serve.h | 15 ++\n> > >> 8 files changed, 380 insertions(+)\n> > >> create mode 100644 Documentation/technical/protocol-v2.txt\n> > >> create mode 100644 builtin/serve.c\n> > >> create mode 100644 serve.c\n> > >> create mode 100644 serve.h\n> > >>\n> > >> diff --git a/.gitignore b/.gitignore\n> > >> index 833ef3b0b..2d0450c26 100644\n> > >> --- a/.gitignore\n> > >> +++ b/.gitignore\n> > >> @@ -140,6 +140,7 @@\n> > >> /git-rm\n> > >> /git-send-email\n> > >> /git-send-pack\n> > >> +/git-serve\n> > >> /git-sh-i18n\n> > >> /git-sh-i18n--envsubst\n> > >> /git-sh-setup\n> > >> diff --git a/Documentation/technical/protocol-v2.txt\n> > >> b/Documentation/technical/protocol-v2.txt\n> > >> new file mode 100644\n> > >> index 000000000..b87ba3816\n> > >> --- /dev/null\n> > >> +++ b/Documentation/technical/protocol-v2.txt\n> > >> @@ -0,0 +1,91 @@\n> > >> + Git Wire Protocol, Version 2\n> > >> +==============================\n> > >> +\n> > >> +This document presents a specification for a version 2 of Git's wire\n> > >> +protocol. Protocol v2 will improve upon v1 in the following ways:\n> > >> +\n> > >> + * Instead of multiple service names, multiple commands will be\n> > >> + supported by a single service.\n> > >> + * Easily extendable as capabilities are moved into their own section\n> > >> + of the protocol, no longer being hidden behind a NUL byte and\n> > >> + limited by the size of a pkt-line (as there will be a single\n> > >> + capability per pkt-line).\n> > >> + * Separate out other information hidden behind NUL bytes (e.g. agent\n> > >> + string as a capability and symrefs can be requested using\n> > >> + 'ls-refs')\n> > >> + * Reference advertisement will be omitted unless explicitly\n> > >> + requested\n> > >> + * ls-refs command to explicitly request some refs\n> > >> +\n> > >> + Detailed Design\n> > >> +=================\n> > >> +\n> > >> +A client can request to speak protocol v2 by sending `version=2` in\n> > >> +the side-channel `GIT_PROTOCOL` in the initial request to the server.\n> > >> +\n> > >> +In protocol v2 communication is command oriented. When first\n> > >> +contacting\n> > >> a\n> > >> +server a list of capabilities will advertised. Some of these\n> > >> capabilities\n> > >> +will be commands which a client can request be executed. Once a\n> > >> +command has completed, a client can reuse the connection and request\n> > >> +that other commands be executed.\n> > >> +\n> > >> + Special Packets\n> > >> +-----------------\n> > >> +\n> > >> +In protocol v2 these special packets will have the following semantics:\n> > >> +\n> > >> + * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n> > >> + * '0001' Delimiter Packet (delim-pkt) - separates sections of a\n> > >> + message\n> > >\n> > >\n> > > Previously, a 0001 pkt-line meant that there was 1 byte of data\n> > > following, right?\n> > \n> > No, the length was including the length field, so 0005 would indicate that\n> > there is one byte following, (+4 bytes of \"0005\" included)\n> > \n> > > Does this change that and/or prevent 1 byte packets? (Not sure if it\n> > > is likely, but the odd-tail of a packfile might get sent in a 0001\n> > > line, right?) Or is it that 0001 is only special during the V2\n> > > negotiation stuff, but not during the packfile transmission?\n> > \n> > 0001 is invalid in the current protocol v0.\n> > \n> > >\n> > > (I'm not against having this delimiter -- I think it is useful, but\n> > > just curious if will cause problems elsewhere.)\n> > >\n> > > Should we also consider increasing the pkt-line limit to 5 hex-digits\n> > > while we're at it ? That would let us have 1MB buffers if that would\n> > > help with large packfiles.\n> > \n> > AFAICT there is a static allocation of one pkt-line (of maximum size), such\n> > that the code can read in a full packet and then process it.\n> > If we'd increase the packet size we'd need the static buffer to be 1MB, which\n> > sounds good for my developer machine. But I suspect it may be too much for\n> > people using git on embedded devices?\n> > \n> > pack files larger than 64k are put into multiple pkt-lines, which is not a big\n> > deal, as the overhead of 4bytes per 64k is negligible.\n> > (also there is progress information in the side channel, which would come in\n> > as a special packet in between real packets, such that every 64k transmitted\n> > you can update your progress meter; Not sure I feel strongly on fewer\n> > progress updates)\n> \n> Can I request, selfishly from my own platform's (NonStop) performance heartache, that we don't require 1Mb? We're not embedded on this platform, but there is an optimized message system packet size down at 50Kb that I would like to stay under. Although above that is no problem, there is a significant cost incurred above that size point. And please make sure xread/xwrite are used in any event.\n\nI think that it would be too much of a change to up to 1MB lines at the\nmoment so I'm planning on leaving it right where it is :)\n\n> \n> > > Granted, we're throttled by the network, so it might not matter.\n> > > Would it be interesting to have a 5 digit prefix with parts of the\n> > > high bits of first digit being flags ?\n> > > Or is this too radical of a change?\n> > \n> > What would the flags be for?\n> > \n> > As an alternative we could put the channel number in one byte, such that we\n> > can have a side channel not just while streaming the pack but all the time.\n> > (Again, not sure if that buys a lot for us)\n> \n> Cheers,\n> Randall\n> \n> -- Brief whoami:\n> NonStop developer since approximately 211288444200000000\n> UNIX developer since approximately 421664400\n> -- In my real life, I talk too much.\n> \n> \n> \n\n-- \nBrandon Williams\n"},{"id":"338046","messageId":"006101d39b9c$77e8b820$67ba2860$@nexbridge.com","threadId":"47524","inReplyTo":"20180201200826.GC138727@google.com","subject":"RE: [PATCH 11/26] serve: introduce git-serve","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-02-01T20:37:19Z","receivedAt":"2018-02-01T20:37:33Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On February 1, 2018 3:08 PM, Brandon Williams wrote:\n> On 02/01, Randall S. Becker wrote:\n> > On February 1, 2018 1:58 PM, Stefan Beller wrote:\n> > > On Thu, Feb 1, 2018 at 10:48 AM, Jeff Hostetler\n> > > \n> > > wrote:\n> > > >\n> > > >\n> > > > On 1/2/2018 7:18 PM, Brandon Williams wrote:\n> > > >>\n> > > >> Introduce git-serve, the base server for protocol version 2.\n> > > >>\n> > > >> Protocol version 2 is intended to be a replacement for Git's\n> > > >> current wire protocol. The intention is that it will be a\n> > > >> simpler, less wasteful protocol which can evolve over time.\n> > > >>\n> > > >> Protocol version 2 improves upon version 1 by eliminating the\n> > > >> initial ref advertisement. In its place a server will export a\n> > > >> list of capabilities and commands which it supports in a\n> > > >> capability advertisement. A client can then request that a\n> > > >> particular command be executed by providing a number of\n> > > >> capabilities and command specific parameters. At the completion\n> > > >> of a command, a client can request that another command be\n> > > >> executed or can terminate the connection by sending a flush packet.\n> > > >>\n> > > >> Signed-off-by: Brandon Williams \n> > > >> ---\n> > > >> .gitignore | 1 +\n> > > >> Documentation/technical/protocol-v2.txt | 91 ++++++++++++\n> > > >> Makefile | 2 +\n> > > >> builtin.h | 1 +\n> > > >> builtin/serve.c | 30 ++++\n> > > >> git.c | 1 +\n> > > >> serve.c | 239\n> > > >> ++++++++++++++++++++++++++++++++\n> > > >> serve.h | 15 ++\n> > > >> 8 files changed, 380 insertions(+)\n> > > >> create mode 100644 Documentation/technical/protocol-v2.txt\n> > > >> create mode 100644 builtin/serve.c\n> > > >> create mode 100644 serve.c\n> > > >> create mode 100644 serve.h\n> > > >>\n> > > >> diff --git a/.gitignore b/.gitignore index 833ef3b0b..2d0450c26\n> > > >> 100644\n> > > >> --- a/.gitignore\n> > > >> +++ b/.gitignore\n> > > >> @@ -140,6 +140,7 @@\n> > > >> /git-rm\n> > > >> /git-send-email\n> > > >> /git-send-pack\n> > > >> +/git-serve\n> > > >> /git-sh-i18n\n> > > >> /git-sh-i18n--envsubst\n> > > >> /git-sh-setup\n> > > >> diff --git a/Documentation/technical/protocol-v2.txt\n> > > >> b/Documentation/technical/protocol-v2.txt\n> > > >> new file mode 100644\n> > > >> index 000000000..b87ba3816\n> > > >> --- /dev/null\n> > > >> +++ b/Documentation/technical/protocol-v2.txt\n> > > >> @@ -0,0 +1,91 @@\n> > > >> + Git Wire Protocol, Version 2\n> > > >> +==============================\n> > > >> +\n> > > >> +This document presents a specification for a version 2 of Git's\n> > > >> +wire protocol. Protocol v2 will improve upon v1 in the following\n> ways:\n> > > >> +\n> > > >> + * Instead of multiple service names, multiple commands will be\n> > > >> + supported by a single service.\n> > > >> + * Easily extendable as capabilities are moved into their own\nsection\n> > > >> + of the protocol, no longer being hidden behind a NUL byte and\n> > > >> + limited by the size of a pkt-line (as there will be a single\n> > > >> + capability per pkt-line).\n> > > >> + * Separate out other information hidden behind NUL bytes (e.g.\n> agent\n> > > >> + string as a capability and symrefs can be requested using\n> > > >> + 'ls-refs')\n> > > >> + * Reference advertisement will be omitted unless explicitly\n> > > >> + requested\n> > > >> + * ls-refs command to explicitly request some refs\n> > > >> +\n> > > >> + Detailed Design\n> > > >> +=================\n> > > >> +\n> > > >> +A client can request to speak protocol v2 by sending `version=2`\n> > > >> +in the side-channel `GIT_PROTOCOL` in the initial request to the\n> server.\n> > > >> +\n> > > >> +In protocol v2 communication is command oriented. When first\n> > > >> +contacting\n> > > >> a\n> > > >> +server a list of capabilities will advertised. Some of these\n> > > >> capabilities\n> > > >> +will be commands which a client can request be executed. Once a\n> > > >> +command has completed, a client can reuse the connection and\n> > > >> +request that other commands be executed.\n> > > >> +\n> > > >> + Special Packets\n> > > >> +-----------------\n> > > >> +\n> > > >> +In protocol v2 these special packets will have the following\n> semantics:\n> > > >> +\n> > > >> + * '0000' Flush Packet (flush-pkt) - indicates the end of a\n> > > >> + message\n> > > >> + * '0001' Delimiter Packet (delim-pkt) - separates sections of\n> > > >> + a message\n> > > >\n> > > >\n> > > > Previously, a 0001 pkt-line meant that there was 1 byte of data\n> > > > following, right?\n> > >\n> > > No, the length was including the length field, so 0005 would\n> > > indicate that there is one byte following, (+4 bytes of \"0005\"\n> > > included)\n> > >\n> > > > Does this change that and/or prevent 1 byte packets? (Not sure if\n> > > > it is likely, but the odd-tail of a packfile might get sent in a\n> > > > 0001 line, right?) Or is it that 0001 is only special during the\n> > > > V2 negotiation stuff, but not during the packfile transmission?\n> > >\n> > > 0001 is invalid in the current protocol v0.\n> > >\n> > > >\n> > > > (I'm not against having this delimiter -- I think it is useful,\n> > > > but just curious if will cause problems elsewhere.)\n> > > >\n> > > > Should we also consider increasing the pkt-line limit to 5\nhex-digits\n> > > > while we're at it ? That would let us have 1MB buffers if that\nwould\n> > > > help with large packfiles.\n> > >\n> > > AFAICT there is a static allocation of one pkt-line (of maximum\n> > > size), such that the code can read in a full packet and then process\nit.\n> > > If we'd increase the packet size we'd need the static buffer to be\n> > > 1MB, which sounds good for my developer machine. But I suspect it\n> > > may be too much for people using git on embedded devices?\n> > >\n> > > pack files larger than 64k are put into multiple pkt-lines, which is\n> > > not a big deal, as the overhead of 4bytes per 64k is negligible.\n> > > (also there is progress information in the side channel, which would\n> > > come in as a special packet in between real packets, such that every\n> > > 64k transmitted you can update your progress meter; Not sure I feel\n> > > strongly on fewer progress updates)\n> >\n> > Can I request, selfishly from my own platform's (NonStop) performance\n> heartache, that we don't require 1Mb? We're not embedded on this\n> platform, but there is an optimized message system packet size down at\n> 50Kb that I would like to stay under. Although above that is no problem,\n> there is a significant cost incurred above that size point. And please\nmake\n> sure xread/xwrite are used in any event.\n> \n> I think that it would be too much of a change to up to 1MB lines at the\n> moment so I'm planning on leaving it right where it is :)\n\nIn for a kilo, in for a tonne. Once we're way up there, it's not a problem\nor much of a difference. :)\n\n> > > > Granted, we're throttled by the network, so it might not matter.\n> > > > Would it be interesting to have a 5 digit prefix with parts of the\n> > > > high bits of first digit being flags ?\n> > > > Or is this too radical of a change?\n> > >\n> > > What would the flags be for?\n> > >\n> > > As an alternative we could put the channel number in one byte, such\n> > > that we can have a side channel not just while streaming the pack but\nall\n> the time.\n> > > (Again, not sure if that buys a lot for us)\n\n"},{"id":"338047","messageId":"CAGZ79kY-95gL1cxOXx7+9OGd6SyRSqwZRYUNv-SCW8Kcn3ZFSw@mail.gmail.com","threadId":"47524","inReplyTo":"006101d39b9c$77e8b820$67ba2860$@nexbridge.com","subject":"Re: [PATCH 11/26] serve: introduce git-serve","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-01T20:50:48Z","receivedAt":"2018-02-01T20:50:59Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Feb 1, 2018 at 12:37 PM, Randall S. Becker\n wrote:\n\n>> I think that it would be too much of a change to up to 1MB lines at the\n>> moment so I'm planning on leaving it right where it is :)\n>\n> In for a kilo, in for a tonne. Once we're way up there, it's not a problem\n> or much of a difference. :)\n\nWhat benefit does a larger buffer have?\n\nI outlined the negatives above (large static buffer, issues with\nprogress meter).\n\nAnd it seems to me that Brandon wants to keep this series as small as possible\nw.r.t. bait for endless discussions and only deliver innovation, that solves the\nimmediate needs. Are there issues with too small buffers? (Can you link to\nthe performance measurements or an analysis?)\n"},{"id":"338122","messageId":"20180202223154.GA187827@google.com","threadId":"47524","inReplyTo":"CAGZ79kb14ciUoyHLAxmh0jGwwPYFVjGayrn9R2f+4+88A8=xsA@mail.gmail.com","subject":"Re: [PATCH v2 13/27] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-02T22:31:54Z","receivedAt":"2018-02-02T22:32:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/26, Stefan Beller wrote:\n> On Thu, Jan 25, 2018 at 3:58 PM, Brandon Williams wrote:\n> \n> > +ls-refs takes in the following parameters wrapped in packet-lines:\n> > +\n> > + symrefs\n> > + In addition to the object pointed by it, show the underlying ref\n> > + pointed by it when showing a symbolic ref.\n> > + peel\n> > + Show peeled tags.\n> \n> Would it make sense to default these two to on, and rather have\n> optional no-symrefs and no-peel ?\n> \n> That would save bandwidth in the default case, I would think.\n\nMaybe? That would save sending those strings for each request\n\n-- \nBrandon Williams\n"},{"id":"338127","messageId":"20180202224454.GB187827@google.com","threadId":"47524","inReplyTo":"815183f7-f449-1286-6dee-7cf6064bc5a1@gmail.com","subject":"Re: [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-02T22:44:54Z","receivedAt":"2018-02-02T22:45:02Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/31, Derrick Stolee wrote:\n> On 1/25/2018 6:58 PM, Brandon Williams wrote:\n> > Introduce protocol_v2, a new value for 'enum protocol_version'.\n> > Subsequent patches will fill in the implementation of protocol_v2.\n> > \n> > Signed-off-by: Brandon Williams \n> > ---\n> > builtin/fetch-pack.c | 3 +++\n> > builtin/receive-pack.c | 6 ++++++\n> > builtin/send-pack.c | 3 +++\n> > builtin/upload-pack.c | 7 +++++++\n> > connect.c | 3 +++\n> > protocol.c | 2 ++\n> > protocol.h | 1 +\n> > remote-curl.c | 3 +++\n> > transport.c | 9 +++++++++\n> > 9 files changed, 37 insertions(+)\n> > \n> > diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n> > index 85d4faf76..f492e8abd 100644\n> > --- a/builtin/fetch-pack.c\n> > +++ b/builtin/fetch-pack.c\n> > @@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> > \t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > \tswitch (discover_version(&reader)) {\n> > +\tcase protocol_v2:\n> > +\t\tdie(\"support for protocol v2 not implemented yet\");\n> > +\t\tbreak;\n> > \tcase protocol_v1:\n> > \tcase protocol_v0:\n> > \t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n> > diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> > index b7ce7c7f5..3656e94fd 100644\n> > --- a/builtin/receive-pack.c\n> > +++ b/builtin/receive-pack.c\n> > @@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> > \t\tunpack_limit = receive_unpack_limit;\n> > \tswitch (determine_protocol_version_server()) {\n> > +\tcase protocol_v2:\n> > +\t\t/*\n> > +\t\t * push support for protocol v2 has not been implemented yet,\n> > +\t\t * so ignore the request to use v2 and fallback to using v0.\n> > +\t\t */\n> > +\t\tbreak;\n> > \tcase protocol_v1:\n> > \t\t/*\n> > \t\t * v1 is just the original protocol with a version string,\n> > diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n> > index 83cb125a6..b5427f75e 100644\n> > --- a/builtin/send-pack.c\n> > +++ b/builtin/send-pack.c\n> > @@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n> > \t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > \tswitch (discover_version(&reader)) {\n> > +\tcase protocol_v2:\n> > +\t\tdie(\"support for protocol v2 not implemented yet\");\n> > +\t\tbreak;\n> > \tcase protocol_v1:\n> > \tcase protocol_v0:\n> > \t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n> > diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\n> > index 2cb5cb35b..8d53e9794 100644\n> > --- a/builtin/upload-pack.c\n> > +++ b/builtin/upload-pack.c\n> > @@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n> > \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n> > \tswitch (determine_protocol_version_server()) {\n> > +\tcase protocol_v2:\n> > +\t\t/*\n> > +\t\t * fetch support for protocol v2 has not been implemented yet,\n> > +\t\t * so ignore the request to use v2 and fallback to using v0.\n> > +\t\t */\n> > +\t\tupload_pack(&opts);\n> > +\t\tbreak;\n> > \tcase protocol_v1:\n> > \t\t/*\n> > \t\t * v1 is just the original protocol with a version string,\n> > diff --git a/connect.c b/connect.c\n> > index db3c9d24c..f2157a821 100644\n> > --- a/connect.c\n> > +++ b/connect.c\n> > @@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)\n> > \t/* Maybe process capabilities here, at least for v2 */\n> > \tswitch (version) {\n> > +\tcase protocol_v2:\n> > +\t\tdie(\"support for protocol v2 not implemented yet\");\n> > +\t\tbreak;\n> > \tcase protocol_v1:\n> > \t\t/* Read the peeked version line */\n> > \t\tpacket_reader_read(reader);\n> > diff --git a/protocol.c b/protocol.c\n> > index 43012b7eb..5e636785d 100644\n> > --- a/protocol.c\n> > +++ b/protocol.c\n> > @@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)\n> > \t\treturn protocol_v0;\n> > \telse if (!strcmp(value, \"1\"))\n> > \t\treturn protocol_v1;\n> > +\telse if (!strcmp(value, \"2\"))\n> > +\t\treturn protocol_v2;\n> > \telse\n> > \t\treturn protocol_unknown_version;\n> > }\n> > diff --git a/protocol.h b/protocol.h\n> > index 1b2bc94a8..2ad35e433 100644\n> > --- a/protocol.h\n> > +++ b/protocol.h\n> > @@ -5,6 +5,7 @@ enum protocol_version {\n> > \tprotocol_unknown_version = -1,\n> > \tprotocol_v0 = 0,\n> > \tprotocol_v1 = 1,\n> > +\tprotocol_v2 = 2,\n> > };\n> > /*\n> > diff --git a/remote-curl.c b/remote-curl.c\n> > index 9f6d07683..dae8a4a48 100644\n> > --- a/remote-curl.c\n> > +++ b/remote-curl.c\n> > @@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n> > \t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > \tswitch (discover_version(&reader)) {\n> > +\tcase protocol_v2:\n> > +\t\tdie(\"support for protocol v2 not implemented yet\");\n> > +\t\tbreak;\n> > \tcase protocol_v1:\n> > \tcase protocol_v0:\n> > \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n> > diff --git a/transport.c b/transport.c\n> > index 2378dcb38..83d9dd1df 100644\n> > --- a/transport.c\n> > +++ b/transport.c\n> > @@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n> > \tdata->version = discover_version(&reader);\n> > \tswitch (data->version) {\n> > +\tcase protocol_v2:\n> > +\t\tdie(\"support for protocol v2 not implemented yet\");\n> > +\t\tbreak;\n> > \tcase protocol_v1:\n> > \tcase protocol_v0:\n> > \t\tget_remote_heads(&reader, &refs,\n> > @@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,\n> > \t\trefs_tmp = get_refs_via_connect(transport, 0);\n> > \tswitch (data->version) {\n> > +\tcase protocol_v2:\n> > +\t\tdie(\"support for protocol v2 not implemented yet\");\n> > +\t\tbreak;\n> > \tcase protocol_v1:\n> > \tcase protocol_v0:\n> > \t\trefs = fetch_pack(&args, data->fd, data->conn,\n> > @@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n> > \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n> > \tswitch (data->version) {\n> > +\tcase protocol_v2:\n> > +\t\tdie(\"support for protocol v2 not implemented yet\");\n> > +\t\tbreak;\n> > \tcase protocol_v1:\n> > \tcase protocol_v0:\n> > \t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n> \n> With a macro approach to version selection, this change becomes simpler in\n> some ways and harder in others.\n> \n> It is simpler in that we can have the macro from the previous commits just\n> fall back to version 0 behavior.\n> \n> It is harder in that this commit would need one of two options:\n> \n> 1. A macro that performs an arbitrary statement when given v2, which would\n> be the die() for these actions not in v2.\n> 2. A macro that clearly states v2 is not supported and calls die() on v2.\n> \n> Here is my simple, untested attempt at a union of these options:\n> \n> #define ON_PROTOCOL_VERSION(version,v0,v2) switch(version) {\\\n> case protocol_v2:\\\n>     (v2);\\\n>     break;\\\n> case protocol_v1:\\\n> case protocol_v0:\\\n>     (v0);\\\n>     break;\\\n> case protocol_unknown_version:\\\n>     BUG(\"unknown protocol version\");\\\n> }\n> #define ON_PROTOCOL_VERSION_V0_FALLBACK(version,v0) switch(version) {\\\n> case protocol_v2:\\\n> case protocol_v1:\\\n> case protocol_v0:\\\n>     (v0);\\\n>     break;\\\n> case protocol_unknown_version:\\\n>     BUG(\"unknown protocol version\");\\\n> }\n> #define ON_PROTOCOL_VERSION_V0_ONLY(version,v0) \\\n>     ON_PROTOCOL_VERSION(version,v0,\\\n>                 BUG(\"support for protocol v2 not implemented yet\"))\n\n\nWhile I understand wanting to isolate the switch statement code, I think\nthat creating such a macro would make reading the code much more\ndifficult (and a pain to get right). Really I don't want to try my hand\nat crafting such a macro :D\n\n-- \nBrandon Williams\n"},{"id":"338217","messageId":"592b2701-de7d-293b-f374-a885bdd827df@gmail.com","threadId":"47524","inReplyTo":"20180202224454.GB187827@google.com","subject":"Re: [PATCH v2 10/27] protocol: introduce enum protocol_version value protocol_v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-05T14:14:59Z","receivedAt":"2018-02-05T14:15:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/2/2018 5:44 PM, Brandon Williams wrote:\n> On 01/31, Derrick Stolee wrote:\n>> On 1/25/2018 6:58 PM, Brandon Williams wrote:\n>>> Introduce protocol_v2, a new value for 'enum protocol_version'.\n>>> Subsequent patches will fill in the implementation of protocol_v2.\n>>>\n>>> Signed-off-by: Brandon Williams \n>>> ---\n>>> builtin/fetch-pack.c | 3 +++\n>>> builtin/receive-pack.c | 6 ++++++\n>>> builtin/send-pack.c | 3 +++\n>>> builtin/upload-pack.c | 7 +++++++\n>>> connect.c | 3 +++\n>>> protocol.c | 2 ++\n>>> protocol.h | 1 +\n>>> remote-curl.c | 3 +++\n>>> transport.c | 9 +++++++++\n>>> 9 files changed, 37 insertions(+)\n>>>\n>>> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n>>> index 85d4faf76..f492e8abd 100644\n>>> --- a/builtin/fetch-pack.c\n>>> +++ b/builtin/fetch-pack.c\n>>> @@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n>>> \t\t\t PACKET_READ_GENTLE_ON_EOF);\n>>> \tswitch (discover_version(&reader)) {\n>>> +\tcase protocol_v2:\n>>> +\t\tdie(\"support for protocol v2 not implemented yet\");\n>>> +\t\tbreak;\n>>> \tcase protocol_v1:\n>>> \tcase protocol_v0:\n>>> \t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n>>> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n>>> index b7ce7c7f5..3656e94fd 100644\n>>> --- a/builtin/receive-pack.c\n>>> +++ b/builtin/receive-pack.c\n>>> @@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n>>> \t\tunpack_limit = receive_unpack_limit;\n>>> \tswitch (determine_protocol_version_server()) {\n>>> +\tcase protocol_v2:\n>>> +\t\t/*\n>>> +\t\t * push support for protocol v2 has not been implemented yet,\n>>> +\t\t * so ignore the request to use v2 and fallback to using v0.\n>>> +\t\t */\n>>> +\t\tbreak;\n>>> \tcase protocol_v1:\n>>> \t\t/*\n>>> \t\t * v1 is just the original protocol with a version string,\n>>> diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n>>> index 83cb125a6..b5427f75e 100644\n>>> --- a/builtin/send-pack.c\n>>> +++ b/builtin/send-pack.c\n>>> @@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n>>> \t\t\t PACKET_READ_GENTLE_ON_EOF);\n>>> \tswitch (discover_version(&reader)) {\n>>> +\tcase protocol_v2:\n>>> +\t\tdie(\"support for protocol v2 not implemented yet\");\n>>> +\t\tbreak;\n>>> \tcase protocol_v1:\n>>> \tcase protocol_v0:\n>>> \t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n>>> diff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\n>>> index 2cb5cb35b..8d53e9794 100644\n>>> --- a/builtin/upload-pack.c\n>>> +++ b/builtin/upload-pack.c\n>>> @@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n>>> \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n>>> \tswitch (determine_protocol_version_server()) {\n>>> +\tcase protocol_v2:\n>>> +\t\t/*\n>>> +\t\t * fetch support for protocol v2 has not been implemented yet,\n>>> +\t\t * so ignore the request to use v2 and fallback to using v0.\n>>> +\t\t */\n>>> +\t\tupload_pack(&opts);\n>>> +\t\tbreak;\n>>> \tcase protocol_v1:\n>>> \t\t/*\n>>> \t\t * v1 is just the original protocol with a version string,\n>>> diff --git a/connect.c b/connect.c\n>>> index db3c9d24c..f2157a821 100644\n>>> --- a/connect.c\n>>> +++ b/connect.c\n>>> @@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)\n>>> \t/* Maybe process capabilities here, at least for v2 */\n>>> \tswitch (version) {\n>>> +\tcase protocol_v2:\n>>> +\t\tdie(\"support for protocol v2 not implemented yet\");\n>>> +\t\tbreak;\n>>> \tcase protocol_v1:\n>>> \t\t/* Read the peeked version line */\n>>> \t\tpacket_reader_read(reader);\n>>> diff --git a/protocol.c b/protocol.c\n>>> index 43012b7eb..5e636785d 100644\n>>> --- a/protocol.c\n>>> +++ b/protocol.c\n>>> @@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)\n>>> \t\treturn protocol_v0;\n>>> \telse if (!strcmp(value, \"1\"))\n>>> \t\treturn protocol_v1;\n>>> +\telse if (!strcmp(value, \"2\"))\n>>> +\t\treturn protocol_v2;\n>>> \telse\n>>> \t\treturn protocol_unknown_version;\n>>> }\n>>> diff --git a/protocol.h b/protocol.h\n>>> index 1b2bc94a8..2ad35e433 100644\n>>> --- a/protocol.h\n>>> +++ b/protocol.h\n>>> @@ -5,6 +5,7 @@ enum protocol_version {\n>>> \tprotocol_unknown_version = -1,\n>>> \tprotocol_v0 = 0,\n>>> \tprotocol_v1 = 1,\n>>> +\tprotocol_v2 = 2,\n>>> };\n>>> /*\n>>> diff --git a/remote-curl.c b/remote-curl.c\n>>> index 9f6d07683..dae8a4a48 100644\n>>> --- a/remote-curl.c\n>>> +++ b/remote-curl.c\n>>> @@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n>>> \t\t\t PACKET_READ_GENTLE_ON_EOF);\n>>> \tswitch (discover_version(&reader)) {\n>>> +\tcase protocol_v2:\n>>> +\t\tdie(\"support for protocol v2 not implemented yet\");\n>>> +\t\tbreak;\n>>> \tcase protocol_v1:\n>>> \tcase protocol_v0:\n>>> \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n>>> diff --git a/transport.c b/transport.c\n>>> index 2378dcb38..83d9dd1df 100644\n>>> --- a/transport.c\n>>> +++ b/transport.c\n>>> @@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n>>> \tdata->version = discover_version(&reader);\n>>> \tswitch (data->version) {\n>>> +\tcase protocol_v2:\n>>> +\t\tdie(\"support for protocol v2 not implemented yet\");\n>>> +\t\tbreak;\n>>> \tcase protocol_v1:\n>>> \tcase protocol_v0:\n>>> \t\tget_remote_heads(&reader, &refs,\n>>> @@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,\n>>> \t\trefs_tmp = get_refs_via_connect(transport, 0);\n>>> \tswitch (data->version) {\n>>> +\tcase protocol_v2:\n>>> +\t\tdie(\"support for protocol v2 not implemented yet\");\n>>> +\t\tbreak;\n>>> \tcase protocol_v1:\n>>> \tcase protocol_v0:\n>>> \t\trefs = fetch_pack(&args, data->fd, data->conn,\n>>> @@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n>>> \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n>>> \tswitch (data->version) {\n>>> +\tcase protocol_v2:\n>>> +\t\tdie(\"support for protocol v2 not implemented yet\");\n>>> +\t\tbreak;\n>>> \tcase protocol_v1:\n>>> \tcase protocol_v0:\n>>> \t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n>> With a macro approach to version selection, this change becomes simpler in\n>> some ways and harder in others.\n>>\n>> It is simpler in that we can have the macro from the previous commits just\n>> fall back to version 0 behavior.\n>>\n>> It is harder in that this commit would need one of two options:\n>>\n>> 1. A macro that performs an arbitrary statement when given v2, which would\n>> be the die() for these actions not in v2.\n>> 2. A macro that clearly states v2 is not supported and calls die() on v2.\n>>\n>> Here is my simple, untested attempt at a union of these options:\n>>\n>> #define ON_PROTOCOL_VERSION(version,v0,v2) switch(version) {\\\n>> case protocol_v2:\\\n>>     (v2);\\\n>>     break;\\\n>> case protocol_v1:\\\n>> case protocol_v0:\\\n>>     (v0);\\\n>>     break;\\\n>> case protocol_unknown_version:\\\n>>     BUG(\"unknown protocol version\");\\\n>> }\n>> #define ON_PROTOCOL_VERSION_V0_FALLBACK(version,v0) switch(version) {\\\n>> case protocol_v2:\\\n>> case protocol_v1:\\\n>> case protocol_v0:\\\n>>     (v0);\\\n>>     break;\\\n>> case protocol_unknown_version:\\\n>>     BUG(\"unknown protocol version\");\\\n>> }\n>> #define ON_PROTOCOL_VERSION_V0_ONLY(version,v0) \\\n>>     ON_PROTOCOL_VERSION(version,v0,\\\n>>                 BUG(\"support for protocol v2 not implemented yet\"))\n>\n> While I understand wanting to isolate the switch statement code, I think\n> that creating such a macro would make reading the code much more\n> difficult (and a pain to get right). Really I don't want to try my hand\n> at crafting such a macro :D\n>\n\nSounds good. You're right that the macro approach is more likely to be \nused incorrectly.\n\n-Stolee\n"},{"id":"338550","messageId":"20180207005522.GA222904@google.com","threadId":"47524","inReplyTo":"e2fb1a05-3cbd-5226-7796-c5b89c9d3955@jeffhostetler.com","subject":"Re: [PATCH 12/26] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T00:55:22Z","receivedAt":"2018-02-07T00:55:30Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/01, Jeff Hostetler wrote:\n> \n> \n> On 1/2/2018 7:18 PM, Brandon Williams wrote:\n> > Introduce the ls-refs server command. In protocol v2, the ls-refs\n> > command is used to request the ref advertisement from the server. Since\n> > it is a command which can be requested (as opposed to mandatory in v1),\n> > a client can sent a number of parameters in its request to limit the ref\n> > advertisement based on provided ref-patterns.\n> > \n> > Signed-off-by: Brandon Williams \n> > ---\n> > Documentation/technical/protocol-v2.txt | 26 +++++++++\n> > Makefile | 1 +\n> > ls-refs.c | 97 +++++++++++++++++++++++++++++++++\n> > ls-refs.h | 9 +++\n> > serve.c | 2 +\n> > 5 files changed, 135 insertions(+)\n> > create mode 100644 ls-refs.c\n> > create mode 100644 ls-refs.h\n> > \n> > diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> > index b87ba3816..5f4d0e719 100644\n> > --- a/Documentation/technical/protocol-v2.txt\n> > +++ b/Documentation/technical/protocol-v2.txt\n> > @@ -89,3 +89,29 @@ terminate the connection.\n> > Commands are the core actions that a client wants to perform (fetch, push,\n> > etc). Each command will be provided with a list capabilities and\n> > arguments as requested by a client.\n> > +\n> > + Ls-refs\n> > +---------\n> > +\n> > +Ls-refs is the command used to request a reference advertisement in v2.\n> > +Unlike the current reference advertisement, ls-refs takes in parameters\n> > +which can be used to limit the refs sent from the server.\n> > +\n> > +Ls-ref takes in the following parameters wraped in packet-lines:\n> > +\n> > + symrefs: In addition to the object pointed by it, show the underlying\n> > +\t ref pointed by it when showing a symbolic ref.\n> > + peel: Show peeled tags.\n> > + ref-pattern : When specified, only references matching the\n> > +\t\t\t given patterns are displayed.\n> > +\n> > +The output of ls-refs is as follows:\n> > +\n> > + output = *ref\n> > +\t flush-pkt\n> > + ref = PKT-LINE((tip | peeled) LF)\n> > + tip = obj-id SP refname (SP symref-target)\n> > + peeled = obj-id SP refname \"^{}\"\n> > +\n> > + symref = PKT-LINE(\"symref\" SP symbolic-ref SP resolved-ref LF)\n> > + shallow = PKT-LINE(\"shallow\" SP obj-id LF)\n> \n> Do you want to talk about ordering requirements on this?\n> I think packed-refs has one, but I'm not sure it matters here\n> where the client or server sorts it.\n> \n> Are there any provisions for compressing the renames, like in the\n> reftable spec or in index-v4 ?\n\nNot currently but it would be rather easy to just add a feature to\nls-refs to transmit the resultant list of refs into something like\nreftable. So this is something that can be added later.\n\n> \n> It doesn't need to be in the initial version. Just asking. We could\n> always add a \"ls-refs-2\" command that builds upon this.\n> \n> Thanks,\n> Jeff\n\n-- \nBrandon Williams\n"},{"id":"338551","messageId":"20180207005820.GB222904@google.com","threadId":"47524","inReplyTo":"56044237-4086-faeb-730e-807858a082a8@gmail.com","subject":"Re: [PATCH v2 00/27] protocol version 2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T00:58:20Z","receivedAt":"2018-02-07T00:58:28Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/31, Derrick Stolee wrote:\n> Sorry for chiming in with mostly nitpicks so late since sending this\n> version. Mostly, I tried to read it to see if I could understand the scope\n> of the patch and how this code worked before. It looks very polished, so I\n> the nits were the best I could do.\n> \n> On 1/25/2018 6:58 PM, Brandon Williams wrote:\n> > Changes in v2:\n> > * Added documentation for fetch\n> > * changes #defines for state variables to be enums\n> > * couple code changes to pkt-line functions and documentation\n> > * Added unit tests for the git-serve binary as well as for ls-refs\n> \n> I'm a fan of more unit-level testing, and I think that will be more\n> important as we go on with these multiple configuration options.\n> \n> > Areas for improvement\n> > * Push isn't implemented, right now this is ok because if v2 is requested the\n> > server can just default to v0. Before this can be merged we may want to\n> > change how the client request a new protocol, and not allow for sending\n> > \"version=2\" when pushing even though the user has it configured. Or maybe\n> > its fine to just have an older client who doesn't understand how to push\n> > (and request v2) to die if the server tries to speak v2 at it.\n> > \n> > Fixing this essentially would just require piping through a bit more\n> > information to the function which ultimately runs connect (for both builtins\n> > and remote-curl)\n> \n> Definitely save push for a later patch. Getting 'fetch' online did require\n> 'ls-refs' at the same time. Future reviews will be easier when adding one\n> command at a time.\n> \n> > \n> > * I want to make sure that the docs are well written before this gets merged\n> > so I'm hoping that someone can do a through review on the docs themselves to\n> > make sure they are clear.\n> \n> I made a comment in the docs about the architectural changes. While I think\n> a discussion on that topic would be valuable, I'm not sure that's the point\n> of the document (i.e. documenting what v2 does versus selling the value of\n> the patch). I thought the docs were clear for how the commands work.\n> \n> > * Right now there is a capability 'stateless-rpc' which essentially makes sure\n> > that a server command completes after a single round (this is to make sure\n> > http works cleanly). After talking with some folks it may make more sense\n> > to just have v2 be stateless in nature so that all commands terminate after\n> > a single round trip. This makes things a bit easier if a server wants to\n> > have ssh just be a proxy for http.\n> > \n> > One potential thing would be to flip this so that by default the protocol is\n> > stateless and if a server/command has a state-full mode that can be\n> > implemented as a capability at a later point. Thoughts?\n> \n> At minimum, all commands should be designed with a \"stateless first\"\n> philosophy since a large number of users communicate via HTTP[S] and any\n> decisions that make stateless communication painful should be rejected.\n\nI agree with this and my next version will run with this philosophy in\nmind (v2 will be stateless by default).\n\n> \n> > * Shallow repositories and shallow clones aren't supported yet. I'm working\n> > on it and it can be either added to v2 by default if people think it needs\n> > to be in there from the start, or we can add it as a capability at a later\n> > point.\n> \n> I'm happy to say the following:\n> \n> 1. Shallow repositories should not be used for servers, since they cannot\n> service all requests.\n> \n> 2. Since v2 has easy capability features, I'm happy to leave shallow for\n> later. We will want to verify that a shallow clone command reverts to v1.\n> \n> \n> I fetched bw/protocol-v2 with tip 13c70148, built, set 'protocol.version=2'\n> in the config, and tested fetches against GitHub and VSTS just as a\n> compatibility test. Everything worked just fine.\n> \n> Is there an easy way to test the existing test suite for clone and fetch\n> using protocol v2 to make sure there are no regressions with\n> protocol.version=2 in the config?\n\nYes there already exist interop tests for testing the addition of\nrequesting a new protocol at //t/interop/i5700-protocol-transition.sh\n\n> \n> Thanks,\n> -Stolee\n\n-- \nBrandon Williams\n"},{"id":"338552","messageId":"20180207011312.189834-1-bmwill@google.com","threadId":"47524","inReplyTo":"20180125235838.138135-1-bmwill@google.com","subject":"[PATCH v3 00/35] protocol version 2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:37Z","receivedAt":"2018-02-07T01:13:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v3:\n * There were some comments about how the protocol should be designed\n stateless first. I've made this change and instead of having to\n supply the `stateless-rpc=true` capability to force stateless\n behavior, the protocol just requires all commands to be stateless.\n \n * Added some patches towards the end of the series to force the client\n to not request to use protocol v2 when pushing (even if configured to\n use v2). This is to ease the roll-out process of a push command in\n protocol v2. This way when servers gain the ability to accept\n pushing in v2 (and they start responding using v2 when requests are\n sent to the git-receive-pack endpoint) that clients who still don't\n understand how to push using v2 won't request to use v2 and then die\n when they recognize that the server does indeed know how to accept a\n push under v2.\n\n * I implemented the `shallow` feature for fetch. This feature\n encapsulates the existing functionality of all the shallow/deepen\n capabilities in v0. So now a server can process shallow requests.\n\n * Various other small tweaks that I can't remember :)\n\nAfter all of that I think the series is in a pretty good state, baring\nany more critical reviewing feedback.\n\nThanks!\n\nBrandon Williams (35):\n pkt-line: introduce packet_read_with_status\n pkt-line: introduce struct packet_reader\n pkt-line: add delim packet support\n upload-pack: convert to a builtin\n upload-pack: factor out processing lines\n transport: use get_refs_via_connect to get refs\n connect: convert get_remote_heads to use struct packet_reader\n connect: discover protocol version outside of get_remote_heads\n transport: store protocol version\n protocol: introduce enum protocol_version value protocol_v2\n test-pkt-line: introduce a packet-line test helper\n serve: introduce git-serve\n ls-refs: introduce ls-refs server command\n connect: request remote refs using v2\n transport: convert get_refs_list to take a list of ref patterns\n transport: convert transport_get_remote_refs to take a list of ref\n patterns\n ls-remote: pass ref patterns when requesting a remote's refs\n fetch: pass ref patterns when fetching\n push: pass ref patterns when pushing\n upload-pack: introduce fetch server command\n fetch-pack: perform a fetch using v2\n upload-pack: support shallow requests\n fetch-pack: support shallow requests\n connect: refactor git_connect to only get the protocol version once\n connect: don't request v2 when pushing\n transport-helper: remove name parameter\n transport-helper: refactor process_connect_service\n transport-helper: introduce stateless-connect\n pkt-line: add packet_buf_write_len function\n remote-curl: create copy of the service name\n remote-curl: store the protocol version the server responded with\n http: allow providing extra headers for http requests\n http: don't always add Git-Protocol header\n remote-curl: implement stateless-connect command\n remote-curl: don't request v2 when pushing\n\n .gitignore | 1 +\n Documentation/technical/protocol-v2.txt | 338 +++++++++++++++++\n Makefile | 7 +-\n builtin.h | 2 +\n builtin/clone.c | 2 +-\n builtin/fetch-pack.c | 21 +-\n builtin/fetch.c | 14 +-\n builtin/ls-remote.c | 7 +-\n builtin/receive-pack.c | 6 +\n builtin/remote.c | 2 +-\n builtin/send-pack.c | 20 +-\n builtin/serve.c | 30 ++\n builtin/upload-pack.c | 74 ++++\n connect.c | 352 +++++++++++++-----\n connect.h | 7 +\n fetch-pack.c | 319 +++++++++++++++-\n fetch-pack.h | 4 +-\n git.c | 2 +\n http.c | 25 +-\n http.h | 2 +\n ls-refs.c | 96 +++++\n ls-refs.h | 9 +\n pkt-line.c | 149 +++++++-\n pkt-line.h | 77 ++++\n protocol.c | 2 +\n protocol.h | 1 +\n remote-curl.c | 257 ++++++++++++-\n remote.h | 9 +-\n serve.c | 260 +++++++++++++\n serve.h | 15 +\n t/helper/test-pkt-line.c | 64 ++++\n t/t5701-git-serve.sh | 176 +++++++++\n t/t5702-protocol-v2.sh | 239 ++++++++++++\n transport-helper.c | 84 +++--\n transport-internal.h | 4 +-\n transport.c | 116 ++++--\n transport.h | 9 +-\n upload-pack.c | 625 ++++++++++++++++++++++++--------\n upload-pack.h | 21 ++\n 39 files changed, 3088 insertions(+), 360 deletions(-)\n create mode 100644 Documentation/technical/protocol-v2.txt\n create mode 100644 builtin/serve.c\n create mode 100644 builtin/upload-pack.c\n create mode 100644 ls-refs.c\n create mode 100644 ls-refs.h\n create mode 100644 serve.c\n create mode 100644 serve.h\n create mode 100644 t/helper/test-pkt-line.c\n create mode 100755 t/t5701-git-serve.sh\n create mode 100755 t/t5702-protocol-v2.sh\n create mode 100644 upload-pack.h\n\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338553","messageId":"20180207011312.189834-2-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 01/35] pkt-line: introduce packet_read_with_status","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:38Z","receivedAt":"2018-02-07T01:13:26Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The current pkt-line API encodes the status of a pkt-line read in the\nlength of the read content. An error is indicated with '-1', a flush\nwith '0' (which can be confusing since a return value of '0' can also\nindicate an empty pkt-line), and a positive integer for the length of\nthe read content otherwise. This doesn't leave much room for allowing\nthe addition of additional special packets in the future.\n\nTo solve this introduce 'packet_read_with_status()' which reads a packet\nand returns the status of the read encoded as an 'enum packet_status'\ntype. This allows for easily identifying between special and normal\npackets as well as errors. It also enables easily adding a new special\npacket in the future.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 57 +++++++++++++++++++++++++++++++++++++++++++--------------\n pkt-line.h | 15 +++++++++++++++\n 2 files changed, 58 insertions(+), 14 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 2827ca772..af0d2430f 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -280,28 +280,33 @@ static int packet_length(const char *linelen)\n \treturn (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);\n }\n \n-int packet_read(int fd, char **src_buf, size_t *src_len,\n-\t\tchar *buffer, unsigned size, int options)\n+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n+\t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n+\t\t\t\t\t\tint options)\n {\n-\tint len, ret;\n+\tint len;\n \tchar linelen[4];\n \n-\tret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);\n-\tif (ret < 0)\n-\t\treturn ret;\n+\tif (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)\n+\t\treturn PACKET_READ_EOF;\n+\n \tlen = packet_length(linelen);\n-\tif (len < 0)\n+\n+\tif (len < 0) {\n \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n-\tif (!len) {\n+\t} else if (!len) {\n \t\tpacket_trace(\"0000\", 4, 0);\n-\t\treturn 0;\n+\t\treturn PACKET_READ_FLUSH;\n+\t} else if (len < 4) {\n+\t\tdie(\"protocol error: bad line length %d\", len);\n \t}\n+\n \tlen -= 4;\n-\tif (len >= size)\n+\tif ((unsigned)len >= size)\n \t\tdie(\"protocol error: bad line length %d\", len);\n-\tret = get_packet_data(fd, src_buf, src_len, buffer, len, options);\n-\tif (ret < 0)\n-\t\treturn ret;\n+\n+\tif (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)\n+\t\treturn PACKET_READ_EOF;\n \n \tif ((options & PACKET_READ_CHOMP_NEWLINE) &&\n \t len && buffer[len-1] == '\\n')\n@@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,\n \n \tbuffer[len] = 0;\n \tpacket_trace(buffer, len, 0);\n-\treturn len;\n+\t*pktlen = len;\n+\treturn PACKET_READ_NORMAL;\n+}\n+\n+int packet_read(int fd, char **src_buffer, size_t *src_len,\n+\t\tchar *buffer, unsigned size, int options)\n+{\n+\tenum packet_read_status status;\n+\tint pktlen;\n+\n+\tstatus = packet_read_with_status(fd, src_buffer, src_len,\n+\t\t\t\t\t buffer, size, &pktlen,\n+\t\t\t\t\t options);\n+\tswitch (status) {\n+\tcase PACKET_READ_EOF:\n+\t\tpktlen = -1;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\tbreak;\n+\tcase PACKET_READ_FLUSH:\n+\t\tpktlen = 0;\n+\t\tbreak;\n+\t}\n+\n+\treturn pktlen;\n }\n \n static char *packet_read_line_generic(int fd,\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3dad583e2..06c468927 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n int packet_read(int fd, char **src_buffer, size_t *src_len, char\n \t\t*buffer, unsigned size, int options);\n \n+/*\n+ * Read a packetized line into a buffer like the 'packet_read()' function but\n+ * returns an 'enum packet_read_status' which indicates the status of the read.\n+ * The number of bytes read will be assigined to *pktlen if the status of the\n+ * read was 'PACKET_READ_NORMAL'.\n+ */\n+enum packet_read_status {\n+\tPACKET_READ_EOF = -1,\n+\tPACKET_READ_NORMAL,\n+\tPACKET_READ_FLUSH,\n+};\n+enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n+\t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n+\t\t\t\t\t\tint options);\n+\n /*\n * Convenience wrapper for packet_read that is not gentle, and sets the\n * CHOMP_NEWLINE option. The return value is NULL for a flush packet,\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338554","messageId":"20180207011312.189834-3-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 02/35] pkt-line: introduce struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:39Z","receivedAt":"2018-02-07T01:13:30Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Sometimes it is advantageous to be able to peek the next packet line\nwithout consuming it (e.g. to be able to determine the protocol version\na server is speaking). In order to do that introduce 'struct\npacket_reader' which is an abstraction around the normal packet reading\nlogic. This enables a caller to be able to peek a single line at a time\nusing 'packet_reader_peek()' and having a caller consume a line by\ncalling 'packet_reader_read()'.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n pkt-line.h | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 117 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex af0d2430f..4fc9ad4b0 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -406,3 +406,62 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)\n \t}\n \treturn sb_out->len - orig_len;\n }\n+\n+/* Packet Reader Functions */\n+void packet_reader_init(struct packet_reader *reader, int fd,\n+\t\t\tchar *src_buffer, size_t src_len,\n+\t\t\tint options)\n+{\n+\tmemset(reader, 0, sizeof(*reader));\n+\n+\treader->fd = fd;\n+\treader->src_buffer = src_buffer;\n+\treader->src_len = src_len;\n+\treader->buffer = packet_buffer;\n+\treader->buffer_size = sizeof(packet_buffer);\n+\treader->options = options;\n+}\n+\n+enum packet_read_status packet_reader_read(struct packet_reader *reader)\n+{\n+\tif (reader->line_peeked) {\n+\t\treader->line_peeked = 0;\n+\t\treturn reader->status;\n+\t}\n+\n+\treader->status = packet_read_with_status(reader->fd,\n+\t\t\t\t\t\t &reader->src_buffer,\n+\t\t\t\t\t\t &reader->src_len,\n+\t\t\t\t\t\t reader->buffer,\n+\t\t\t\t\t\t reader->buffer_size,\n+\t\t\t\t\t\t &reader->pktlen,\n+\t\t\t\t\t\t reader->options);\n+\n+\tswitch (reader->status) {\n+\tcase PACKET_READ_EOF:\n+\t\treader->pktlen = -1;\n+\t\treader->line = NULL;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\treader->line = reader->buffer;\n+\t\tbreak;\n+\tcase PACKET_READ_FLUSH:\n+\t\treader->pktlen = 0;\n+\t\treader->line = NULL;\n+\t\tbreak;\n+\t}\n+\n+\treturn reader->status;\n+}\n+\n+enum packet_read_status packet_reader_peek(struct packet_reader *reader)\n+{\n+\t/* Only allow peeking a single line */\n+\tif (reader->line_peeked)\n+\t\treturn reader->status;\n+\n+\t/* Peek a line by reading it and setting peeked flag */\n+\tpacket_reader_read(reader);\n+\treader->line_peeked = 1;\n+\treturn reader->status;\n+}\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 06c468927..7d9f0e537 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n */\n ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);\n \n+struct packet_reader {\n+\t/* source file descriptor */\n+\tint fd;\n+\n+\t/* source buffer and its size */\n+\tchar *src_buffer;\n+\tsize_t src_len;\n+\n+\t/* buffer that pkt-lines are read into and its size */\n+\tchar *buffer;\n+\tunsigned buffer_size;\n+\n+\t/* options to be used during reads */\n+\tint options;\n+\n+\t/* status of the last read */\n+\tenum packet_read_status status;\n+\n+\t/* length of data read during the last read */\n+\tint pktlen;\n+\n+\t/* the last line read */\n+\tconst char *line;\n+\n+\t/* indicates if a line has been peeked */\n+\tint line_peeked;\n+};\n+\n+/*\n+ * Initialize a 'struct packet_reader' object which is an\n+ * abstraction around the 'packet_read_with_status()' function.\n+ */\n+extern void packet_reader_init(struct packet_reader *reader, int fd,\n+\t\t\t char *src_buffer, size_t src_len,\n+\t\t\t int options);\n+\n+/*\n+ * Perform a packet read and return the status of the read.\n+ * The values of 'pktlen' and 'line' are updated based on the status of the\n+ * read as follows:\n+ *\n+ * PACKET_READ_ERROR: 'pktlen' is set to '-1' and 'line' is set to NULL\n+ * PACKET_READ_NORMAL: 'pktlen' is set to the number of bytes read\n+ *\t\t 'line' is set to point at the read line\n+ * PACKET_READ_FLUSH: 'pktlen' is set to '0' and 'line' is set to NULL\n+ */\n+extern enum packet_read_status packet_reader_read(struct packet_reader *reader);\n+\n+/*\n+ * Peek the next packet line without consuming it and return the status.\n+ * The next call to 'packet_reader_read()' will perform a read of the same line\n+ * that was peeked, consuming the line.\n+ *\n+ * Peeking multiple times without calling 'packet_reader_read()' will return\n+ * the same result.\n+ */\n+extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);\n+\n #define DEFAULT_PACKET_MAX 1000\n #define LARGE_PACKET_MAX 65520\n #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338555","messageId":"20180207011312.189834-5-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:41Z","receivedAt":"2018-02-07T01:13:32Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to allow for code sharing with the server-side of fetch in\nprotocol-v2 convert upload-pack to be a builtin.\n\nSigned-off-by: Brandon Williams \n---\n Makefile | 3 +-\n builtin.h | 1 +\n builtin/upload-pack.c | 67 +++++++++++++++++++++++++++++++\n git.c | 1 +\n upload-pack.c | 107 ++++++++++++--------------------------------------\n upload-pack.h | 13 ++++++\n 6 files changed, 109 insertions(+), 83 deletions(-)\n create mode 100644 builtin/upload-pack.c\n create mode 100644 upload-pack.h\n\ndiff --git a/Makefile b/Makefile\nindex 1a9b23b67..b7ccc05fa 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -639,7 +639,6 @@ PROGRAM_OBJS += imap-send.o\n PROGRAM_OBJS += sh-i18n--envsubst.o\n PROGRAM_OBJS += shell.o\n PROGRAM_OBJS += show-index.o\n-PROGRAM_OBJS += upload-pack.o\n PROGRAM_OBJS += remote-testsvn.o\n \n # Binary suffix, set to .exe for Windows builds\n@@ -909,6 +908,7 @@ LIB_OBJS += tree-diff.o\n LIB_OBJS += tree.o\n LIB_OBJS += tree-walk.o\n LIB_OBJS += unpack-trees.o\n+LIB_OBJS += upload-pack.o\n LIB_OBJS += url.o\n LIB_OBJS += urlmatch.o\n LIB_OBJS += usage.o\n@@ -1026,6 +1026,7 @@ BUILTIN_OBJS += builtin/update-index.o\n BUILTIN_OBJS += builtin/update-ref.o\n BUILTIN_OBJS += builtin/update-server-info.o\n BUILTIN_OBJS += builtin/upload-archive.o\n+BUILTIN_OBJS += builtin/upload-pack.o\n BUILTIN_OBJS += builtin/var.o\n BUILTIN_OBJS += builtin/verify-commit.o\n BUILTIN_OBJS += builtin/verify-pack.o\ndiff --git a/builtin.h b/builtin.h\nindex 42378f3aa..f332a1257 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -231,6 +231,7 @@ extern int cmd_update_ref(int argc, const char **argv, const char *prefix);\n extern int cmd_update_server_info(int argc, const char **argv, const char *prefix);\n extern int cmd_upload_archive(int argc, const char **argv, const char *prefix);\n extern int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix);\n+extern int cmd_upload_pack(int argc, const char **argv, const char *prefix);\n extern int cmd_var(int argc, const char **argv, const char *prefix);\n extern int cmd_verify_commit(int argc, const char **argv, const char *prefix);\n extern int cmd_verify_tag(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\nnew file mode 100644\nindex 000000000..2cb5cb35b\n--- /dev/null\n+++ b/builtin/upload-pack.c\n@@ -0,0 +1,67 @@\n+#include \"cache.h\"\n+#include \"builtin.h\"\n+#include \"exec_cmd.h\"\n+#include \"pkt-line.h\"\n+#include \"parse-options.h\"\n+#include \"protocol.h\"\n+#include \"upload-pack.h\"\n+\n+static const char * const upload_pack_usage[] = {\n+\tN_(\"git upload-pack [] \"),\n+\tNULL\n+};\n+\n+int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *dir;\n+\tint strict = 0;\n+\tstruct upload_pack_options opts = { 0 };\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n+\t\t\t N_(\"quit after a single request/response exchange\")),\n+\t\tOPT_BOOL(0, \"advertise-refs\", &opts.advertise_refs,\n+\t\t\t N_(\"exit immediately after initial ref advertisement\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict,\n+\t\t\t N_(\"do not try /.git/ if is no Git directory\")),\n+\t\tOPT_INTEGER(0, \"timeout\", &opts.timeout,\n+\t\t\t N_(\"interrupt transfer after seconds of inactivity\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tpacket_trace_identity(\"upload-pack\");\n+\tcheck_replace_refs = 0;\n+\n+\targc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);\n+\n+\tif (argc != 1)\n+\t\tusage_with_options(upload_pack_usage, options);\n+\n+\tif (opts.timeout)\n+\t\topts.daemon_mode = 1;\n+\n+\tsetup_path();\n+\n+\tdir = argv[0];\n+\n+\tif (!enter_repo(dir, strict))\n+\t\tdie(\"'%s' does not appear to be a git repository\", dir);\n+\n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\t\tif (opts.advertise_refs || !opts.stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\n+\t\t/* fallthrough */\n+\tcase protocol_v0:\n+\t\tupload_pack(&opts);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex c870b9719..f71073dc8 100644\n--- a/git.c\n+++ b/git.c\n@@ -478,6 +478,7 @@ static struct cmd_struct commands[] = {\n \t{ \"update-server-info\", cmd_update_server_info, RUN_SETUP },\n \t{ \"upload-archive\", cmd_upload_archive },\n \t{ \"upload-archive--writer\", cmd_upload_archive_writer },\n+\t{ \"upload-pack\", cmd_upload_pack },\n \t{ \"var\", cmd_var, RUN_SETUP_GENTLY },\n \t{ \"verify-commit\", cmd_verify_commit, RUN_SETUP },\n \t{ \"verify-pack\", cmd_verify_pack },\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d5de18127..2ad73a98b 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -6,7 +6,6 @@\n #include \"tag.h\"\n #include \"object.h\"\n #include \"commit.h\"\n-#include \"exec_cmd.h\"\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n@@ -15,15 +14,10 @@\n #include \"sigchain.h\"\n #include \"version.h\"\n #include \"string-list.h\"\n-#include \"parse-options.h\"\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n #include \"protocol.h\"\n-\n-static const char * const upload_pack_usage[] = {\n-\tN_(\"git upload-pack [] \"),\n-\tNULL\n-};\n+#include \"upload-pack.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -61,7 +55,6 @@ static int keepalive = 5;\n * otherwise maximum packet size (up to 65520 bytes).\n */\n static int use_sideband;\n-static int advertise_refs;\n static int stateless_rpc;\n static const char *pack_objects_hook;\n \n@@ -977,33 +970,6 @@ static int find_symref(const char *refname, const struct object_id *oid,\n \treturn 0;\n }\n \n-static void upload_pack(void)\n-{\n-\tstruct string_list symref = STRING_LIST_INIT_DUP;\n-\n-\thead_ref_namespaced(find_symref, &symref);\n-\n-\tif (advertise_refs || !stateless_rpc) {\n-\t\treset_timeout();\n-\t\thead_ref_namespaced(send_ref, &symref);\n-\t\tfor_each_namespaced_ref(send_ref, &symref);\n-\t\tadvertise_shallow_grafts(1);\n-\t\tpacket_flush(1);\n-\t} else {\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n-\t}\n-\tstring_list_clear(&symref, 1);\n-\tif (advertise_refs)\n-\t\treturn;\n-\n-\treceive_needs();\n-\tif (want_obj.nr) {\n-\t\tget_common_commits();\n-\t\tcreate_pack_file();\n-\t}\n-}\n-\n static int upload_pack_config(const char *var, const char *value, void *unused)\n {\n \tif (!strcmp(\"uploadpack.allowtipsha1inwant\", var)) {\n@@ -1032,58 +998,35 @@ static int upload_pack_config(const char *var, const char *value, void *unused)\n \treturn parse_hide_refs_config(var, value, \"uploadpack\");\n }\n \n-int cmd_main(int argc, const char **argv)\n+void upload_pack(struct upload_pack_options *options)\n {\n-\tconst char *dir;\n-\tint strict = 0;\n-\tstruct option options[] = {\n-\t\tOPT_BOOL(0, \"stateless-rpc\", &stateless_rpc,\n-\t\t\t N_(\"quit after a single request/response exchange\")),\n-\t\tOPT_BOOL(0, \"advertise-refs\", &advertise_refs,\n-\t\t\t N_(\"exit immediately after initial ref advertisement\")),\n-\t\tOPT_BOOL(0, \"strict\", &strict,\n-\t\t\t N_(\"do not try /.git/ if is no Git directory\")),\n-\t\tOPT_INTEGER(0, \"timeout\", &timeout,\n-\t\t\t N_(\"interrupt transfer after seconds of inactivity\")),\n-\t\tOPT_END()\n-\t};\n-\n-\tpacket_trace_identity(\"upload-pack\");\n-\tcheck_replace_refs = 0;\n-\n-\targc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);\n-\n-\tif (argc != 1)\n-\t\tusage_with_options(upload_pack_usage, options);\n-\n-\tif (timeout)\n-\t\tdaemon_mode = 1;\n-\n-\tsetup_path();\n-\n-\tdir = argv[0];\n+\tstruct string_list symref = STRING_LIST_INIT_DUP;\n \n-\tif (!enter_repo(dir, strict))\n-\t\tdie(\"'%s' does not appear to be a git repository\", dir);\n+\tstateless_rpc = options->stateless_rpc;\n+\ttimeout = options->timeout;\n+\tdaemon_mode = options->daemon_mode;\n \n \tgit_config(upload_pack_config, NULL);\n \n-\tswitch (determine_protocol_version_server()) {\n-\tcase protocol_v1:\n-\t\t/*\n-\t\t * v1 is just the original protocol with a version string,\n-\t\t * so just fall through after writing the version string.\n-\t\t */\n-\t\tif (advertise_refs || !stateless_rpc)\n-\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n-\n-\t\t/* fallthrough */\n-\tcase protocol_v0:\n-\t\tupload_pack();\n-\t\tbreak;\n-\tcase protocol_unknown_version:\n-\t\tBUG(\"unknown protocol version\");\n+\thead_ref_namespaced(find_symref, &symref);\n+\n+\tif (options->advertise_refs || !stateless_rpc) {\n+\t\treset_timeout();\n+\t\thead_ref_namespaced(send_ref, &symref);\n+\t\tfor_each_namespaced_ref(send_ref, &symref);\n+\t\tadvertise_shallow_grafts(1);\n+\t\tpacket_flush(1);\n+\t} else {\n+\t\thead_ref_namespaced(check_ref, NULL);\n+\t\tfor_each_namespaced_ref(check_ref, NULL);\n \t}\n+\tstring_list_clear(&symref, 1);\n+\tif (options->advertise_refs)\n+\t\treturn;\n \n-\treturn 0;\n+\treceive_needs();\n+\tif (want_obj.nr) {\n+\t\tget_common_commits();\n+\t\tcreate_pack_file();\n+\t}\n }\ndiff --git a/upload-pack.h b/upload-pack.h\nnew file mode 100644\nindex 000000000..a71e4dc7e\n--- /dev/null\n+++ b/upload-pack.h\n@@ -0,0 +1,13 @@\n+#ifndef UPLOAD_PACK_H\n+#define UPLOAD_PACK_H\n+\n+struct upload_pack_options {\n+\tint stateless_rpc;\n+\tint advertise_refs;\n+\tunsigned int timeout;\n+\tint daemon_mode;\n+};\n+\n+void upload_pack(struct upload_pack_options *options);\n+\n+#endif /* UPLOAD_PACK_H */\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338556","messageId":"20180207011312.189834-4-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 03/35] pkt-line: add delim packet support","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:40Z","receivedAt":"2018-02-07T01:13:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"One of the design goals of protocol-v2 is to improve the semantics of\nflush packets. Currently in protocol-v1, flush packets are used both to\nindicate a break in a list of packet lines as well as an indication that\none side has finished speaking. This makes it particularly difficult\nto implement proxies as a proxy would need to completely understand git\nprotocol instead of simply looking for a flush packet.\n\nTo do this, introduce the special deliminator packet '0001'. A delim\npacket can then be used as a deliminator between lists of packet lines\nwhile flush packets can be reserved to indicate the end of a response.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 17 +++++++++++++++++\n pkt-line.h | 3 +++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 4fc9ad4b0..726e109ca 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -91,6 +91,12 @@ void packet_flush(int fd)\n \twrite_or_die(fd, \"0000\", 4);\n }\n \n+void packet_delim(int fd)\n+{\n+\tpacket_trace(\"0001\", 4, 1);\n+\twrite_or_die(fd, \"0001\", 4);\n+}\n+\n int packet_flush_gently(int fd)\n {\n \tpacket_trace(\"0000\", 4, 1);\n@@ -105,6 +111,12 @@ void packet_buf_flush(struct strbuf *buf)\n \tstrbuf_add(buf, \"0000\", 4);\n }\n \n+void packet_buf_delim(struct strbuf *buf)\n+{\n+\tpacket_trace(\"0001\", 4, 1);\n+\tstrbuf_add(buf, \"0001\", 4);\n+}\n+\n static void set_packet_header(char *buf, const int size)\n {\n \tstatic char hexchar[] = \"0123456789abcdef\";\n@@ -297,6 +309,9 @@ enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_\n \t} else if (!len) {\n \t\tpacket_trace(\"0000\", 4, 0);\n \t\treturn PACKET_READ_FLUSH;\n+\t} else if (len == 1) {\n+\t\tpacket_trace(\"0001\", 4, 0);\n+\t\treturn PACKET_READ_DELIM;\n \t} else if (len < 4) {\n \t\tdie(\"protocol error: bad line length %d\", len);\n \t}\n@@ -333,6 +348,7 @@ int packet_read(int fd, char **src_buffer, size_t *src_len,\n \t\tbreak;\n \tcase PACKET_READ_NORMAL:\n \t\tbreak;\n+\tcase PACKET_READ_DELIM:\n \tcase PACKET_READ_FLUSH:\n \t\tpktlen = 0;\n \t\tbreak;\n@@ -445,6 +461,7 @@ enum packet_read_status packet_reader_read(struct packet_reader *reader)\n \tcase PACKET_READ_NORMAL:\n \t\treader->line = reader->buffer;\n \t\tbreak;\n+\tcase PACKET_READ_DELIM:\n \tcase PACKET_READ_FLUSH:\n \t\treader->pktlen = 0;\n \t\treader->line = NULL;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 7d9f0e537..16fe8bdbf 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -20,8 +20,10 @@\n * side can't, we stay with pure read/write interfaces.\n */\n void packet_flush(int fd);\n+void packet_delim(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n+void packet_buf_delim(struct strbuf *buf);\n void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n@@ -75,6 +77,7 @@ enum packet_read_status {\n \tPACKET_READ_EOF = -1,\n \tPACKET_READ_NORMAL,\n \tPACKET_READ_FLUSH,\n+\tPACKET_READ_DELIM,\n };\n enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n \t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338557","messageId":"20180207011312.189834-6-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 05/35] upload-pack: factor out processing lines","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:42Z","receivedAt":"2018-02-07T01:13:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Factor out the logic for processing shallow, deepen, deepen_since, and\ndeepen_not lines into their own functions to simplify the\n'receive_needs()' function in addition to making it easier to reuse some\nof this logic when implementing protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n upload-pack.c | 113 ++++++++++++++++++++++++++++++++++++++--------------------\n 1 file changed, 74 insertions(+), 39 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 2ad73a98b..1e8a9e1ca 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,\n \tpacket_flush(1);\n }\n \n+static int process_shallow(const char *line, struct object_array *shallows)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"shallow \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tstruct object *object;\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"invalid shallow line: %s\", line);\n+\t\tobject = parse_object(&oid);\n+\t\tif (!object)\n+\t\t\treturn 1;\n+\t\tif (object->type != OBJ_COMMIT)\n+\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n+\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n+\t\t\tobject->flags |= CLIENT_SHALLOW;\n+\t\t\tadd_object_array(object, NULL, shallows);\n+\t\t}\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_deepen(const char *line, int *depth)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\t*depth = (int)strtol(arg, &end, 0);\n+\t\tif (!end || *end || *depth <= 0)\n+\t\t\tdie(\"Invalid deepen: %s\", line);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen-since \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\t*deepen_since = parse_timestamp(arg, &end, 0);\n+\t\tif (!end || *end || !deepen_since ||\n+\t\t /* revisions.c's max_age -1 is special */\n+\t\t *deepen_since == -1)\n+\t\t\tdie(\"Invalid deepen-since: %s\", line);\n+\t\t*deepen_rev_list = 1;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen-not \", &arg)) {\n+\t\tchar *ref = NULL;\n+\t\tstruct object_id oid;\n+\t\tif (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n+\t\t\tdie(\"git upload-pack: ambiguous deepen-not: %s\", line);\n+\t\tstring_list_append(deepen_not, ref);\n+\t\tfree(ref);\n+\t\t*deepen_rev_list = 1;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static void receive_needs(void)\n {\n \tstruct object_array shallows = OBJECT_ARRAY_INIT;\n@@ -745,49 +814,15 @@ static void receive_needs(void)\n \t\tif (!line)\n \t\t\tbreak;\n \n-\t\tif (skip_prefix(line, \"shallow \", &arg)) {\n-\t\t\tstruct object_id oid;\n-\t\t\tstruct object *object;\n-\t\t\tif (get_oid_hex(arg, &oid))\n-\t\t\t\tdie(\"invalid shallow line: %s\", line);\n-\t\t\tobject = parse_object(&oid);\n-\t\t\tif (!object)\n-\t\t\t\tcontinue;\n-\t\t\tif (object->type != OBJ_COMMIT)\n-\t\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n-\t\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n-\t\t\t\tobject->flags |= CLIENT_SHALLOW;\n-\t\t\t\tadd_object_array(object, NULL, &shallows);\n-\t\t\t}\n+\t\tif (process_shallow(line, &shallows))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen \", &arg)) {\n-\t\t\tchar *end = NULL;\n-\t\t\tdepth = strtol(arg, &end, 0);\n-\t\t\tif (!end || *end || depth <= 0)\n-\t\t\t\tdie(\"Invalid deepen: %s\", line);\n+\t\tif (process_deepen(line, &depth))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen-since \", &arg)) {\n-\t\t\tchar *end = NULL;\n-\t\t\tdeepen_since = parse_timestamp(arg, &end, 0);\n-\t\t\tif (!end || *end || !deepen_since ||\n-\t\t\t /* revisions.c's max_age -1 is special */\n-\t\t\t deepen_since == -1)\n-\t\t\t\tdie(\"Invalid deepen-since: %s\", line);\n-\t\t\tdeepen_rev_list = 1;\n+\t\tif (process_deepen_since(line, &deepen_since, &deepen_rev_list))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen-not \", &arg)) {\n-\t\t\tchar *ref = NULL;\n-\t\t\tstruct object_id oid;\n-\t\t\tif (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n-\t\t\t\tdie(\"git upload-pack: ambiguous deepen-not: %s\", line);\n-\t\t\tstring_list_append(&deepen_not, ref);\n-\t\t\tfree(ref);\n-\t\t\tdeepen_rev_list = 1;\n+\t\tif (process_deepen_not(line, &deepen_not, &deepen_rev_list))\n \t\t\tcontinue;\n-\t\t}\n+\n \t\tif (!skip_prefix(line, \"want \", &arg) ||\n \t\t get_oid_hex(arg, &oid_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338558","messageId":"20180207011312.189834-7-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 06/35] transport: use get_refs_via_connect to get refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:43Z","receivedAt":"2018-02-07T01:13:40Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Remove code duplication and use the existing 'get_refs_via_connect()'\nfunction to retrieve a remote's heads in 'fetch_refs_via_pack()' and\n'git_transport_push()'.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 18 ++++--------------\n 1 file changed, 4 insertions(+), 14 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex fc802260f..8e8779096 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n \n-\tif (!data->got_remote_heads) {\n-\t\tconnect_setup(transport, 0);\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL, &data->shallow);\n-\t\tdata->got_remote_heads = 1;\n-\t}\n+\tif (!data->got_remote_heads)\n+\t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n \trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n@@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tstruct send_pack_args args;\n \tint ret;\n \n-\tif (!data->got_remote_heads) {\n-\t\tstruct ref *tmp_refs;\n-\t\tconnect_setup(transport, 1);\n-\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n-\t\t\t\t NULL, &data->shallow);\n-\t\tdata->got_remote_heads = 1;\n-\t}\n+\tif (!data->got_remote_heads)\n+\t\tget_refs_via_connect(transport, 1);\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338559","messageId":"20180207011312.189834-9-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 08/35] connect: discover protocol version outside of get_remote_heads","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:45Z","receivedAt":"2018-02-07T01:13:44Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to prepare for the addition of protocol_v2 push the protocol\nversion discovery outside of 'get_remote_heads()'. This will allow for\nkeeping the logic for processing the reference advertisement for\nprotocol_v1 and protocol_v0 separate from the logic for protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 16 +++++++++++++++-\n builtin/send-pack.c | 17 +++++++++++++++--\n connect.c | 27 ++++++++++-----------------\n connect.h | 3 +++\n remote-curl.c | 20 ++++++++++++++++++--\n remote.h | 5 +++--\n transport.c | 24 +++++++++++++++++++-----\n 7 files changed, 83 insertions(+), 29 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 366b9d13f..85d4faf76 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -4,6 +4,7 @@\n #include \"remote.h\"\n #include \"connect.h\"\n #include \"sha1-array.h\"\n+#include \"protocol.h\"\n \n static const char fetch_pack_usage[] =\n \"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] \"\n@@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tstruct fetch_pack_args args;\n \tstruct oid_array shallow = OID_ARRAY_INIT;\n \tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n+\tstruct packet_reader reader;\n \n \tpacket_trace_identity(\"fetch-pack\");\n \n@@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\tif (!conn)\n \t\t\treturn args.diag_url ? 0 : 1;\n \t}\n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n+\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n \t\t\t &shallow, pack_lockfile_ptr);\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex fc4f0bb5f..83cb125a6 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -14,6 +14,7 @@\n #include \"sha1-array.h\"\n #include \"gpg-interface.h\"\n #include \"gettext.h\"\n+#include \"protocol.h\"\n \n static const char * const send_pack_usage[] = {\n \tN_(\"git send-pack [--all | --mirror] [--dry-run] [--force] \"\n@@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tstruct packet_reader reader;\n \n \tstruct option options[] = {\n \t\tOPT__VERBOSITY(&verbose),\n@@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n-\t\t\t &extra_have, &shallow);\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n+\t\t\t\t &extra_have, &shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/connect.c b/connect.c\nindex 00e90075c..db3c9d24c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n-static enum protocol_version discover_version(struct packet_reader *reader)\n+enum protocol_version discover_version(struct packet_reader *reader)\n {\n \tenum protocol_version version = protocol_unknown_version;\n \n@@ -234,7 +234,7 @@ enum get_remote_heads_state {\n /*\n * Read all the refs from the other end\n */\n-struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n+struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t struct ref **list, unsigned int flags,\n \t\t\t struct oid_array *extra_have,\n \t\t\t struct oid_array *shallow_points)\n@@ -242,24 +242,17 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \tstruct ref **orig_list = list;\n \tint len = 0;\n \tenum get_remote_heads_state state = EXPECTING_FIRST_REF;\n-\tstruct packet_reader reader;\n \tconst char *arg;\n \n-\tpacket_reader_init(&reader, in, src_buf, src_len,\n-\t\t\t PACKET_READ_CHOMP_NEWLINE |\n-\t\t\t PACKET_READ_GENTLE_ON_EOF);\n-\n-\tdiscover_version(&reader);\n-\n \t*list = NULL;\n \n \twhile (state != EXPECTING_DONE) {\n-\t\tswitch (packet_reader_read(&reader)) {\n+\t\tswitch (packet_reader_read(reader)) {\n \t\tcase PACKET_READ_EOF:\n \t\t\tdie_initial_contact(1);\n \t\tcase PACKET_READ_NORMAL:\n-\t\t\tlen = reader.pktlen;\n-\t\t\tif (len > 4 && skip_prefix(reader.line, \"ERR \", &arg))\n+\t\t\tlen = reader->pktlen;\n+\t\t\tif (len > 4 && skip_prefix(reader->line, \"ERR \", &arg))\n \t\t\t\tdie(\"remote error: %s\", arg);\n \t\t\tbreak;\n \t\tcase PACKET_READ_FLUSH:\n@@ -271,22 +264,22 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \n \t\tswitch (state) {\n \t\tcase EXPECTING_FIRST_REF:\n-\t\t\tprocess_capabilities(reader.line, &len);\n-\t\t\tif (process_dummy_ref(reader.line)) {\n+\t\t\tprocess_capabilities(reader->line, &len);\n+\t\t\tif (process_dummy_ref(reader->line)) {\n \t\t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tstate = EXPECTING_REF;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_REF:\n-\t\t\tif (process_ref(reader.line, len, &list, flags, extra_have))\n+\t\t\tif (process_ref(reader->line, len, &list, flags, extra_have))\n \t\t\t\tbreak;\n \t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_SHALLOW:\n-\t\t\tif (process_shallow(reader.line, len, shallow_points))\n+\t\t\tif (process_shallow(reader->line, len, shallow_points))\n \t\t\t\tbreak;\n-\t\t\tdie(\"protocol error: unexpected '%s'\", reader.line);\n+\t\t\tdie(\"protocol error: unexpected '%s'\", reader->line);\n \t\tcase EXPECTING_DONE:\n \t\t\tbreak;\n \t\t}\ndiff --git a/connect.h b/connect.h\nindex 01f14cdf3..cdb8979dc 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);\n extern const char *server_feature_value(const char *feature, int *len_ret);\n extern int url_is_local_not_ssh(const char *url);\n \n+struct packet_reader;\n+extern enum protocol_version discover_version(struct packet_reader *reader);\n+\n #endif\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 0053b0954..9f6d07683 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"remote.h\"\n+#include \"connect.h\"\n #include \"strbuf.h\"\n #include \"walker.h\"\n #include \"http.h\"\n@@ -13,6 +14,7 @@\n #include \"credential.h\"\n #include \"sha1-array.h\"\n #include \"send-pack.h\"\n+#include \"protocol.h\"\n \n static struct remote *remote;\n /* always ends with a trailing slash */\n@@ -176,8 +178,22 @@ static struct discovery *last_discovery;\n static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n-\tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n+\tstruct packet_reader reader;\n+\n+\tpacket_reader_init(&reader, -1, heads->buf, heads->len,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n+\t\t\t\t NULL, &heads->shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \treturn list;\n }\n \ndiff --git a/remote.h b/remote.h\nindex 1f6611be2..2016461df 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n void free_refs(struct ref *ref);\n \n struct oid_array;\n-extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n+struct packet_reader;\n+extern struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t\t struct ref **list, unsigned int flags,\n \t\t\t\t struct oid_array *extra_have,\n-\t\t\t\t struct oid_array *shallow);\n+\t\t\t\t struct oid_array *shallow_points);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\ndiff --git a/transport.c b/transport.c\nindex 8e8779096..63c3dbab9 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -18,6 +18,7 @@\n #include \"sha1-array.h\"\n #include \"sigchain.h\"\n #include \"transport-internal.h\"\n+#include \"protocol.h\"\n \n static void set_upstreams(struct transport *transport, struct ref *refs,\n \tint pretend)\n@@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)\n static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n {\n \tstruct git_transport_data *data = transport->data;\n-\tstruct ref *refs;\n+\tstruct ref *refs = NULL;\n+\tstruct packet_reader reader;\n \n \tconnect_setup(transport, for_push);\n-\tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0,\n-\t\t\t &data->extra_have,\n-\t\t\t &data->shallow);\n+\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &refs,\n+\t\t\t\t for_push ? REF_NORMAL : 0,\n+\t\t\t\t &data->extra_have,\n+\t\t\t\t &data->shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338560","messageId":"20180207011312.189834-10-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 09/35] transport: store protocol version","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:46Z","receivedAt":"2018-02-07T01:13:45Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Once protocol_v2 is introduced requesting a fetch or a push will need to\nbe handled differently depending on the protocol version. Store the\nprotocol version the server is speaking in 'struct git_transport_data'\nand use it to determine what to do in the case of a fetch or a push.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 35 ++++++++++++++++++++++++++---------\n 1 file changed, 26 insertions(+), 9 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex 63c3dbab9..2378dcb38 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -118,6 +118,7 @@ struct git_transport_data {\n \tstruct child_process *conn;\n \tint fd[2];\n \tunsigned got_remote_heads : 1;\n+\tenum protocol_version version;\n \tstruct oid_array extra_have;\n \tstruct oid_array shallow;\n };\n@@ -200,7 +201,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n-\tswitch (discover_version(&reader)) {\n+\tdata->version = discover_version(&reader);\n+\tswitch (data->version) {\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &refs,\n@@ -221,7 +223,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n {\n \tint ret = 0;\n \tstruct git_transport_data *data = transport->data;\n-\tstruct ref *refs;\n+\tstruct ref *refs = NULL;\n \tchar *dest = xstrdup(transport->url);\n \tstruct fetch_pack_args args;\n \tstruct ref *refs_tmp = NULL;\n@@ -247,10 +249,18 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tif (!data->got_remote_heads)\n \t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n-\trefs = fetch_pack(&args, data->fd, data->conn,\n-\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n-\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n-\t\t\t &transport->pack_lockfile);\n+\tswitch (data->version) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\trefs = fetch_pack(&args, data->fd, data->conn,\n+\t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n+\t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n+\t\t\t\t &transport->pack_lockfile);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \tclose(data->fd[0]);\n \tclose(data->fd[1]);\n \tif (finish_connect(data->conn))\n@@ -549,7 +559,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n {\n \tstruct git_transport_data *data = transport->data;\n \tstruct send_pack_args args;\n-\tint ret;\n+\tint ret = 0;\n \n \tif (!data->got_remote_heads)\n \t\tget_refs_via_connect(transport, 1);\n@@ -574,8 +584,15 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \telse\n \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n \n-\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n-\t\t\t&data->extra_have);\n+\tswitch (data->version) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n+\t\t\t\t&data->extra_have);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \tclose(data->fd[1]);\n \tclose(data->fd[0]);\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338561","messageId":"20180207011312.189834-11-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 10/35] protocol: introduce enum protocol_version value protocol_v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:47Z","receivedAt":"2018-02-07T01:13:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce protocol_v2, a new value for 'enum protocol_version'.\nSubsequent patches will fill in the implementation of protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 3 +++\n builtin/receive-pack.c | 6 ++++++\n builtin/send-pack.c | 3 +++\n builtin/upload-pack.c | 7 +++++++\n connect.c | 3 +++\n protocol.c | 2 ++\n protocol.h | 1 +\n remote-curl.c | 3 +++\n transport.c | 9 +++++++++\n 9 files changed, 37 insertions(+)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 85d4faf76..f492e8abd 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex b7ce7c7f5..3656e94fd 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tunpack_limit = receive_unpack_limit;\n \n \tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v2:\n+\t\t/*\n+\t\t * push support for protocol v2 has not been implemented yet,\n+\t\t * so ignore the request to use v2 and fallback to using v0.\n+\t\t */\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 83cb125a6..b5427f75e 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\ndiff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\nindex 2cb5cb35b..8d53e9794 100644\n--- a/builtin/upload-pack.c\n+++ b/builtin/upload-pack.c\n@@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n \tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v2:\n+\t\t/*\n+\t\t * fetch support for protocol v2 has not been implemented yet,\n+\t\t * so ignore the request to use v2 and fallback to using v0.\n+\t\t */\n+\t\tupload_pack(&opts);\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\ndiff --git a/connect.c b/connect.c\nindex db3c9d24c..f2157a821 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -84,6 +84,9 @@ enum protocol_version discover_version(struct packet_reader *reader)\n \n \t/* Maybe process capabilities here, at least for v2 */\n \tswitch (version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/* Read the peeked version line */\n \t\tpacket_reader_read(reader);\ndiff --git a/protocol.c b/protocol.c\nindex 43012b7eb..5e636785d 100644\n--- a/protocol.c\n+++ b/protocol.c\n@@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)\n \t\treturn protocol_v0;\n \telse if (!strcmp(value, \"1\"))\n \t\treturn protocol_v1;\n+\telse if (!strcmp(value, \"2\"))\n+\t\treturn protocol_v2;\n \telse\n \t\treturn protocol_unknown_version;\n }\ndiff --git a/protocol.h b/protocol.h\nindex 1b2bc94a8..2ad35e433 100644\n--- a/protocol.h\n+++ b/protocol.h\n@@ -5,6 +5,7 @@ enum protocol_version {\n \tprotocol_unknown_version = -1,\n \tprotocol_v0 = 0,\n \tprotocol_v1 = 1,\n+\tprotocol_v2 = 2,\n };\n \n /*\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 9f6d07683..dae8a4a48 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\ndiff --git a/transport.c b/transport.c\nindex 2378dcb38..83d9dd1df 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &refs,\n@@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\trefs = fetch_pack(&args, data->fd, data->conn,\n@@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n \n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338562","messageId":"20180207011312.189834-12-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 11/35] test-pkt-line: introduce a packet-line test helper","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:48Z","receivedAt":"2018-02-07T01:13:48Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce a packet-line test helper which can either pack or unpack an\ninput stream into packet-lines and writes out the result to stdout.\n\nSigned-off-by: Brandon Williams \n---\n Makefile | 1 +\n t/helper/test-pkt-line.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 65 insertions(+)\n create mode 100644 t/helper/test-pkt-line.c\n\ndiff --git a/Makefile b/Makefile\nindex b7ccc05fa..3b849c060 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -669,6 +669,7 @@ TEST_PROGRAMS_NEED_X += test-mktemp\n TEST_PROGRAMS_NEED_X += test-online-cpus\n TEST_PROGRAMS_NEED_X += test-parse-options\n TEST_PROGRAMS_NEED_X += test-path-utils\n+TEST_PROGRAMS_NEED_X += test-pkt-line\n TEST_PROGRAMS_NEED_X += test-prio-queue\n TEST_PROGRAMS_NEED_X += test-read-cache\n TEST_PROGRAMS_NEED_X += test-write-cache\ndiff --git a/t/helper/test-pkt-line.c b/t/helper/test-pkt-line.c\nnew file mode 100644\nindex 000000000..0f19e53c7\n--- /dev/null\n+++ b/t/helper/test-pkt-line.c\n@@ -0,0 +1,64 @@\n+#include \"pkt-line.h\"\n+\n+static void pack_line(const char *line)\n+{\n+\tif (!strcmp(line, \"0000\") || !strcmp(line, \"0000\\n\"))\n+\t\tpacket_flush(1);\n+\telse if (!strcmp(line, \"0001\") || !strcmp(line, \"0001\\n\"))\n+\t\tpacket_delim(1);\n+\telse\n+\t\tpacket_write_fmt(1, \"%s\", line);\n+}\n+\n+static void pack(int argc, const char **argv)\n+{\n+\tif (argc) { /* read from argv */\n+\t\tint i;\n+\t\tfor (i = 0; i < argc; i++)\n+\t\t\tpack_line(argv[i]);\n+\t} else { /* read from stdin */\n+\t\tchar line[LARGE_PACKET_MAX];\n+\t\twhile (fgets(line, sizeof(line), stdin)) {\n+\t\t\tpack_line(line);\n+\t\t}\n+\t}\n+}\n+\n+static void unpack(void)\n+{\n+\tstruct packet_reader reader;\n+\tpacket_reader_init(&reader, 0, NULL, 0,\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (packet_reader_read(&reader) != PACKET_READ_EOF) {\n+\t\tswitch (reader.status) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tprintf(\"%s\\n\", reader.line);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tprintf(\"0000\\n\");\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tprintf(\"0001\\n\");\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tif (argc < 2)\n+\t\tdie(\"too few arguments\");\n+\n+\tif (!strcmp(argv[1], \"pack\"))\n+\t\tpack(argc - 2, argv + 2);\n+\telse if (!strcmp(argv[1], \"unpack\"))\n+\t\tunpack();\n+\telse\n+\t\tdie(\"invalid argument '%s'\", argv[1]);\n+\n+\treturn 0;\n+}\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338563","messageId":"20180207011312.189834-13-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 12/35] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:49Z","receivedAt":"2018-02-07T01:13:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce git-serve, the base server for protocol version 2.\n\nProtocol version 2 is intended to be a replacement for Git's current\nwire protocol. The intention is that it will be a simpler, less\nwasteful protocol which can evolve over time.\n\nProtocol version 2 improves upon version 1 by eliminating the initial\nref advertisement. In its place a server will export a list of\ncapabilities and commands which it supports in a capability\nadvertisement. A client can then request that a particular command be\nexecuted by providing a number of capabilities and command specific\nparameters. At the completion of a command, a client can request that\nanother command be executed or can terminate the connection by sending a\nflush packet.\n\nSigned-off-by: Brandon Williams \n---\n .gitignore | 1 +\n Documentation/technical/protocol-v2.txt | 114 +++++++++++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/serve.c | 30 ++++\n git.c | 1 +\n serve.c | 250 ++++++++++++++++++++++++++++++++\n serve.h | 15 ++\n t/t5701-git-serve.sh | 60 ++++++++\n 9 files changed, 474 insertions(+)\n create mode 100644 Documentation/technical/protocol-v2.txt\n create mode 100644 builtin/serve.c\n create mode 100644 serve.c\n create mode 100644 serve.h\n create mode 100755 t/t5701-git-serve.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 833ef3b0b..2d0450c26 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -140,6 +140,7 @@\n /git-rm\n /git-send-email\n /git-send-pack\n+/git-serve\n /git-sh-i18n\n /git-sh-i18n--envsubst\n /git-sh-setup\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nnew file mode 100644\nindex 000000000..f87372f9b\n--- /dev/null\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -0,0 +1,114 @@\n+ Git Wire Protocol, Version 2\n+==============================\n+\n+This document presents a specification for a version 2 of Git's wire\n+protocol. Protocol v2 will improve upon v1 in the following ways:\n+\n+ * Instead of multiple service names, multiple commands will be\n+ supported by a single service\n+ * Easily extendable as capabilities are moved into their own section\n+ of the protocol, no longer being hidden behind a NUL byte and\n+ limited by the size of a pkt-line (as there will be a single\n+ capability per pkt-line)\n+ * Separate out other information hidden behind NUL bytes (e.g. agent\n+ string as a capability and symrefs can be requested using 'ls-refs')\n+ * Reference advertisement will be omitted unless explicitly requested\n+ * ls-refs command to explicitly request some refs\n+ * Designed with http and stateless-rpc in mind. With clear flush\n+ semantics the http remote helper can simply act as a proxy.\n+\n+ Detailed Design\n+=================\n+\n+A client can request to speak protocol v2 by sending `version=2` in the\n+side-channel `GIT_PROTOCOL` in the initial request to the server.\n+\n+In protocol v2 communication is command oriented. When first contacting a\n+server a list of capabilities will advertised. Some of these capabilities\n+will be commands which a client can request be executed. Once a command\n+has completed, a client can reuse the connection and request that other\n+commands be executed.\n+\n+ Special Packets\n+-----------------\n+\n+In protocol v2 these special packets will have the following semantics:\n+\n+ * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n+ * '0001' Delimiter Packet (delim-pkt) - separates sections of a message\n+\n+ Capability Advertisement\n+--------------------------\n+\n+A server which decides to communicate (based on a request from a client)\n+using protocol version 2, notifies the client by sending a version string\n+in its initial response followed by an advertisement of its capabilities.\n+Each capability is a key with an optional value. Clients must ignore all\n+unknown keys. Semantics of unknown values are left to the definition of\n+each key. Some capabilities will describe commands which can be requested\n+to be executed by the client.\n+\n+ capability-advertisement = protocol-version\n+\t\t\t capability-list\n+\t\t\t flush-pkt\n+\n+ protocol-version = PKT-LINE(\"version 2\" LF)\n+ capability-list = *capability\n+ capability = PKT-LINE(key[=value] LF)\n+\n+ key = 1*CHAR\n+ value = 1*CHAR\n+ CHAR = 1*(ALPHA / DIGIT / \"-\" / \"_\")\n+\n+A client then responds to select the command it wants with any particular\n+capabilities or arguments. There is then an optional section where the\n+client can provide any command specific parameters or queries.\n+\n+ command-request = command\n+\t\t capability-list\n+\t\t (command-args)\n+\t\t flush-pkt\n+ command = PKT-LINE(\"command=\" key LF)\n+ command-args = delim-pkt\n+\t\t *arg\n+ arg = 1*CHAR\n+\n+The server will then check to ensure that the client's request is\n+comprised of a valid command as well as valid capabilities which were\n+advertised. If the request is valid the server will then execute the\n+command.\n+\n+When a command has finished a client can either request that another\n+command be executed or can terminate the connection by sending an empty\n+request consisting of just a flush-pkt.\n+\n+ Capabilities\n+~~~~~~~~~~~~~~\n+\n+There are two different types of capabilities: normal capabilities,\n+which can be used to to convey information or alter the behavior of a\n+request, and command capabilities, which are the core actions that a\n+client wants to perform (fetch, push, etc).\n+\n+All commands must only last a single round and be stateless from the\n+perspective of the server side. All state MUST be retained and managed\n+by the client process. This permits simple round-robin load-balancing\n+on the server side, without needing to worry about state management.\n+\n+Clients MUST NOT require state management on the server side in order to\n+function correctly.\n+\n+ agent\n+-------\n+\n+The server can advertise the `agent` capability with a value `X` (in the\n+form `agent=X`) to notify the client that the server is running version\n+`X`. The client may optionally send its own agent string by including\n+the `agent` capability with a value `Y` (in the form `agent=Y`) in its\n+request to the server (but it MUST NOT do so if the server did not\n+advertise the agent capability). The `X` and `Y` strings may contain any\n+printable ASCII characters except space (i.e., the byte range 32 < x <\n+127), and are typically of the form \"package/version\" (e.g.,\n+\"git/1.8.3.1\"). The agent strings are purely informative for statistics\n+and debugging purposes, and MUST NOT be used to programmatically assume\n+the presence or absence of particular features.\ndiff --git a/Makefile b/Makefile\nindex 3b849c060..18c255428 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -881,6 +881,7 @@ LIB_OBJS += revision.o\n LIB_OBJS += run-command.o\n LIB_OBJS += send-pack.o\n LIB_OBJS += sequencer.o\n+LIB_OBJS += serve.o\n LIB_OBJS += server-info.o\n LIB_OBJS += setup.o\n LIB_OBJS += sha1-array.o\n@@ -1014,6 +1015,7 @@ BUILTIN_OBJS += builtin/rev-parse.o\n BUILTIN_OBJS += builtin/revert.o\n BUILTIN_OBJS += builtin/rm.o\n BUILTIN_OBJS += builtin/send-pack.o\n+BUILTIN_OBJS += builtin/serve.o\n BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-ref.o\ndiff --git a/builtin.h b/builtin.h\nindex f332a1257..3f3fdfc28 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -215,6 +215,7 @@ extern int cmd_rev_parse(int argc, const char **argv, const char *prefix);\n extern int cmd_revert(int argc, const char **argv, const char *prefix);\n extern int cmd_rm(int argc, const char **argv, const char *prefix);\n extern int cmd_send_pack(int argc, const char **argv, const char *prefix);\n+extern int cmd_serve(int argc, const char **argv, const char *prefix);\n extern int cmd_shortlog(int argc, const char **argv, const char *prefix);\n extern int cmd_show(int argc, const char **argv, const char *prefix);\n extern int cmd_show_branch(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/serve.c b/builtin/serve.c\nnew file mode 100644\nindex 000000000..d3fd240bb\n--- /dev/null\n+++ b/builtin/serve.c\n@@ -0,0 +1,30 @@\n+#include \"cache.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"serve.h\"\n+\n+static char const * const serve_usage[] = {\n+\tN_(\"git serve []\"),\n+\tNULL\n+};\n+\n+int cmd_serve(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct serve_options opts = SERVE_OPTIONS_INIT;\n+\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n+\t\t\t N_(\"quit after a single request/response exchange\")),\n+\t\tOPT_BOOL(0, \"advertise-capabilities\", &opts.advertise_capabilities,\n+\t\t\t N_(\"exit immediately after advertising capabilities\")),\n+\t\tOPT_END()\n+\t};\n+\n+\t/* ignore all unknown cmdline switches for now */\n+\targc = parse_options(argc, argv, prefix, options, serve_usage,\n+\t\t\t PARSE_OPT_KEEP_DASHDASH |\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tserve(&opts);\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex f71073dc8..f85d682b6 100644\n--- a/git.c\n+++ b/git.c\n@@ -461,6 +461,7 @@ static struct cmd_struct commands[] = {\n \t{ \"revert\", cmd_revert, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"rm\", cmd_rm, RUN_SETUP },\n \t{ \"send-pack\", cmd_send_pack, RUN_SETUP },\n+\t{ \"serve\", cmd_serve, RUN_SETUP },\n \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n \t{ \"show\", cmd_show, RUN_SETUP },\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\ndiff --git a/serve.c b/serve.c\nnew file mode 100644\nindex 000000000..cf23179b9\n--- /dev/null\n+++ b/serve.c\n@@ -0,0 +1,250 @@\n+#include \"cache.h\"\n+#include \"repository.h\"\n+#include \"config.h\"\n+#include \"pkt-line.h\"\n+#include \"version.h\"\n+#include \"argv-array.h\"\n+#include \"serve.h\"\n+\n+static int agent_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, git_user_agent_sanitized());\n+\treturn 1;\n+}\n+\n+struct protocol_capability {\n+\t/*\n+\t * The name of the capability. The server uses this name when\n+\t * advertising this capability, and the client uses this name to\n+\t * specify this capability.\n+\t */\n+\tconst char *name;\n+\n+\t/*\n+\t * Function queried to see if a capability should be advertised.\n+\t * Optionally a value can be specified by adding it to 'value'.\n+\t * If a value is added to 'value', the server will advertise this\n+\t * capability as \"=\" instead of \"\".\n+\t */\n+\tint (*advertise)(struct repository *r, struct strbuf *value);\n+\n+\t/*\n+\t * Function called when a client requests the capability as a command.\n+\t * The command request will be provided to the function via 'keys', the\n+\t * capabilities requested, and 'args', the command specific parameters.\n+\t *\n+\t * This field should be NULL for capabilities which are not commands.\n+\t */\n+\tint (*command)(struct repository *r,\n+\t\t struct argv_array *keys,\n+\t\t struct argv_array *args);\n+};\n+\n+static struct protocol_capability capabilities[] = {\n+\t{ \"agent\", agent_advertise, NULL },\n+};\n+\n+static void advertise_capabilities(void)\n+{\n+\tstruct strbuf capability = STRBUF_INIT;\n+\tstruct strbuf value = STRBUF_INIT;\n+\tint i;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n+\t\tstruct protocol_capability *c = &capabilities[i];\n+\n+\t\tif (c->advertise(the_repository, &value)) {\n+\t\t\tstrbuf_addstr(&capability, c->name);\n+\n+\t\t\tif (value.len) {\n+\t\t\t\tstrbuf_addch(&capability, '=');\n+\t\t\t\tstrbuf_addbuf(&capability, &value);\n+\t\t\t}\n+\n+\t\t\tstrbuf_addch(&capability, '\\n');\n+\t\t\tpacket_write(1, capability.buf, capability.len);\n+\t\t}\n+\n+\t\tstrbuf_reset(&capability);\n+\t\tstrbuf_reset(&value);\n+\t}\n+\n+\tpacket_flush(1);\n+\tstrbuf_release(&capability);\n+\tstrbuf_release(&value);\n+}\n+\n+static struct protocol_capability *get_capability(const char *key)\n+{\n+\tint i;\n+\n+\tif (!key)\n+\t\treturn NULL;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n+\t\tstruct protocol_capability *c = &capabilities[i];\n+\t\tconst char *out;\n+\t\tif (skip_prefix(key, c->name, &out) && (!*out || *out == '='))\n+\t\t\treturn c;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static int is_valid_capability(const char *key)\n+{\n+\tconst struct protocol_capability *c = get_capability(key);\n+\n+\treturn c && c->advertise(the_repository, NULL);\n+}\n+\n+static int is_command(const char *key, struct protocol_capability **command)\n+{\n+\tconst char *out;\n+\n+\tif (skip_prefix(key, \"command=\", &out)) {\n+\t\tstruct protocol_capability *cmd = get_capability(out);\n+\n+\t\tif (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)\n+\t\t\tdie(\"invalid command '%s'\", out);\n+\t\tif (*command)\n+\t\t\tdie(\"command already requested\");\n+\n+\t\t*command = cmd;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int has_capability(const struct argv_array *keys, const char *capability,\n+\t\t const char **value)\n+{\n+\tint i;\n+\tfor (i = 0; i < keys->argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(keys->argv[i], capability, &out) &&\n+\t\t (!*out || *out == '=')) {\n+\t\t\tif (value) {\n+\t\t\t\tif (*out == '=')\n+\t\t\t\t\tout++;\n+\t\t\t\t*value = out;\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+enum request_state {\n+\tPROCESS_REQUEST_KEYS = 0,\n+\tPROCESS_REQUEST_ARGS,\n+\tPROCESS_REQUEST_DONE,\n+};\n+\n+static int process_request(void)\n+{\n+\tenum request_state state = PROCESS_REQUEST_KEYS;\n+\tstruct packet_reader reader;\n+\tstruct argv_array keys = ARGV_ARRAY_INIT;\n+\tstruct argv_array args = ARGV_ARRAY_INIT;\n+\tstruct protocol_capability *command = NULL;\n+\n+\tpacket_reader_init(&reader, 0, NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\t/*\n+\t * Check to see if the client closed their end before sending another\n+\t * request. If so we can terminate the connection.\n+\t */\n+\tif (packet_reader_peek(&reader) == PACKET_READ_EOF)\n+\t\treturn 1;\n+\treader.options = PACKET_READ_CHOMP_NEWLINE;\n+\n+\twhile (state != PROCESS_REQUEST_DONE) {\n+\t\tswitch (packet_reader_read(&reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tBUG(\"Should have already died when seeing EOF\");\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tstate = PROCESS_REQUEST_DONE;\n+\t\t\tcontinue;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tif (state != PROCESS_REQUEST_KEYS)\n+\t\t\t\tdie(\"protocol error\");\n+\t\t\tstate = PROCESS_REQUEST_ARGS;\n+\t\t\t/*\n+\t\t\t * maybe include a check to make sure that a\n+\t\t\t * command/capabilities were given.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (state) {\n+\t\tcase PROCESS_REQUEST_KEYS:\n+\t\t\t/* collect request; a sequence of keys and values */\n+\t\t\tif (is_command(reader.line, &command) ||\n+\t\t\t is_valid_capability(reader.line))\n+\t\t\t\targv_array_push(&keys, reader.line);\n+\t\t\telse\n+\t\t\t\tdie(\"unknown capability '%s'\", reader.line);\n+\t\t\tbreak;\n+\t\tcase PROCESS_REQUEST_ARGS:\n+\t\t\t/* collect arguments for the requested command */\n+\t\t\targv_array_push(&args, reader.line);\n+\t\t\tbreak;\n+\t\tcase PROCESS_REQUEST_DONE:\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If no command and no keys were given then the client wanted to\n+\t * terminate the connection.\n+\t */\n+\tif (!keys.argc && !args.argc)\n+\t\treturn 1;\n+\n+\tif (!command)\n+\t\tdie(\"no command requested\");\n+\n+\tcommand->command(the_repository, &keys, &args);\n+\n+\targv_array_clear(&keys);\n+\targv_array_clear(&args);\n+\treturn 0;\n+}\n+\n+/* Main serve loop for protocol version 2 */\n+void serve(struct serve_options *options)\n+{\n+\tif (options->advertise_capabilities || !options->stateless_rpc) {\n+\t\t/* serve by default supports v2 */\n+\t\tpacket_write_fmt(1, \"version 2\\n\");\n+\n+\t\tadvertise_capabilities();\n+\t\t/*\n+\t\t * If only the list of capabilities was requested exit\n+\t\t * immediately after advertising capabilities\n+\t\t */\n+\t\tif (options->advertise_capabilities)\n+\t\t\treturn;\n+\t}\n+\n+\t/*\n+\t * If stateless-rpc was requested then exit after\n+\t * a single request/response exchange\n+\t */\n+\tif (options->stateless_rpc) {\n+\t\tprocess_request();\n+\t} else {\n+\t\tfor (;;)\n+\t\t\tif (process_request())\n+\t\t\t\tbreak;\n+\t}\n+}\ndiff --git a/serve.h b/serve.h\nnew file mode 100644\nindex 000000000..fe65ba9f4\n--- /dev/null\n+++ b/serve.h\n@@ -0,0 +1,15 @@\n+#ifndef SERVE_H\n+#define SERVE_H\n+\n+struct argv_array;\n+extern int has_capability(const struct argv_array *keys, const char *capability,\n+\t\t\t const char **value);\n+\n+struct serve_options {\n+\tunsigned advertise_capabilities;\n+\tunsigned stateless_rpc;\n+};\n+#define SERVE_OPTIONS_INIT { 0 }\n+extern void serve(struct serve_options *options);\n+\n+#endif /* SERVE_H */\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nnew file mode 100755\nindex 000000000..affbad097\n--- /dev/null\n+++ b/t/t5701-git-serve.sh\n@@ -0,0 +1,60 @@\n+#!/bin/sh\n+\n+test_description='test git-serve and server commands'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'test capability advertisement' '\n+\tcat >expect <<-EOF &&\n+\tversion 2\n+\tagent=git/$(git version | cut -d\" \" -f3)\n+\t0000\n+\tEOF\n+\n+\tgit serve --advertise-capabilities >out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'stateless-rpc flag does not list capabilities' '\n+\t# Empty request\n+\ttest-pkt-line pack >in <<-EOF &&\n+\t0000\n+\tEOF\n+\tgit serve --stateless-rpc >out out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'request invalid capability' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tfoobar\n+\t0000\n+\tEOF\n+\ttest_must_fail git serve --stateless-rpc 2>err in <<-EOF &&\n+\tagent=git/test\n+\t0000\n+\tEOF\n+\ttest_must_fail git serve --stateless-rpc 2>err in <<-EOF &&\n+\tcommand=foo\n+\tagent=git/test\n+\t0000\n+\tEOF\n+\ttest_must_fail git serve --stateless-rpc 2>err \n---\n Documentation/technical/protocol-v2.txt | 32 +++++++++\n Makefile | 1 +\n ls-refs.c | 96 ++++++++++++++++++++++++++\n ls-refs.h | 9 +++\n serve.c | 8 +++\n t/t5701-git-serve.sh | 115 ++++++++++++++++++++++++++++++++\n 6 files changed, 261 insertions(+)\n create mode 100644 ls-refs.c\n create mode 100644 ls-refs.h\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex f87372f9b..ef81df868 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -112,3 +112,35 @@ printable ASCII characters except space (i.e., the byte range 32 < x <\n \"git/1.8.3.1\"). The agent strings are purely informative for statistics\n and debugging purposes, and MUST NOT be used to programmatically assume\n the presence or absence of particular features.\n+\n+ ls-refs\n+---------\n+\n+`ls-refs` is the command used to request a reference advertisement in v2.\n+Unlike the current reference advertisement, ls-refs takes in parameters\n+which can be used to limit the refs sent from the server.\n+\n+Additional features not supported in the base command will be advertised\n+as the value of the command in the capability advertisement in the form\n+of a space separated list of features, e.g. \"=\n+\".\n+\n+ls-refs takes in the following parameters wrapped in packet-lines:\n+\n+ symrefs\n+\tIn addition to the object pointed by it, show the underlying ref\n+\tpointed by it when showing a symbolic ref.\n+ peel\n+\tShow peeled tags.\n+ ref-pattern \n+\tWhen specified, only references matching the one of the provided\n+\tpatterns are displayed.\n+\n+The output of ls-refs is as follows:\n+\n+ output = *ref\n+\t flush-pkt\n+ ref = PKT-LINE(obj-id SP refname *(SP ref-attribute) LF)\n+ ref-attribute = (symref | peeled)\n+ symref = \"symref-target:\" symref-target\n+ peeled = \"peeled:\" obj-id\ndiff --git a/Makefile b/Makefile\nindex 18c255428..e50927cfb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -825,6 +825,7 @@ LIB_OBJS += list-objects-filter-options.o\n LIB_OBJS += ll-merge.o\n LIB_OBJS += lockfile.o\n LIB_OBJS += log-tree.o\n+LIB_OBJS += ls-refs.o\n LIB_OBJS += mailinfo.o\n LIB_OBJS += mailmap.o\n LIB_OBJS += match-trees.o\ndiff --git a/ls-refs.c b/ls-refs.c\nnew file mode 100644\nindex 000000000..70682b4f7\n--- /dev/null\n+++ b/ls-refs.c\n@@ -0,0 +1,96 @@\n+#include \"cache.h\"\n+#include \"repository.h\"\n+#include \"refs.h\"\n+#include \"remote.h\"\n+#include \"argv-array.h\"\n+#include \"ls-refs.h\"\n+#include \"pkt-line.h\"\n+\n+struct ls_refs_data {\n+\tunsigned peel;\n+\tunsigned symrefs;\n+\tstruct argv_array patterns;\n+};\n+\n+/*\n+ * Check if one of the patterns matches the tail part of the ref.\n+ * If no patterns were provided, all refs match.\n+ */\n+static int ref_match(const struct argv_array *patterns, const char *refname)\n+{\n+\tchar *pathbuf;\n+\tint i;\n+\n+\tif (!patterns->argc)\n+\t\treturn 1; /* no restriction */\n+\n+\tpathbuf = xstrfmt(\"/%s\", refname);\n+\tfor (i = 0; i < patterns->argc; i++) {\n+\t\tif (!wildmatch(patterns->argv[i], pathbuf, 0)) {\n+\t\t\tfree(pathbuf);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\tfree(pathbuf);\n+\treturn 0;\n+}\n+\n+static int send_ref(const char *refname, const struct object_id *oid,\n+\t\t int flag, void *cb_data)\n+{\n+\tstruct ls_refs_data *data = cb_data;\n+\tconst char *refname_nons = strip_namespace(refname);\n+\tstruct strbuf refline = STRBUF_INIT;\n+\n+\tif (!ref_match(&data->patterns, refname))\n+\t\treturn 0;\n+\n+\tstrbuf_addf(&refline, \"%s %s\", oid_to_hex(oid), refname_nons);\n+\tif (data->symrefs && flag & REF_ISSYMREF) {\n+\t\tstruct object_id unused;\n+\t\tconst char *symref_target = resolve_ref_unsafe(refname, 0,\n+\t\t\t\t\t\t\t &unused,\n+\t\t\t\t\t\t\t &flag);\n+\n+\t\tif (!symref_target)\n+\t\t\tdie(\"'%s' is a symref but it is not?\", refname);\n+\n+\t\tstrbuf_addf(&refline, \" symref-target:%s\", symref_target);\n+\t}\n+\n+\tif (data->peel) {\n+\t\tstruct object_id peeled;\n+\t\tif (!peel_ref(refname, &peeled))\n+\t\t\tstrbuf_addf(&refline, \" peeled:%s\", oid_to_hex(&peeled));\n+\t}\n+\n+\tstrbuf_addch(&refline, '\\n');\n+\tpacket_write(1, refline.buf, refline.len);\n+\n+\tstrbuf_release(&refline);\n+\treturn 0;\n+}\n+\n+int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)\n+{\n+\tint i;\n+\tstruct ls_refs_data data = { 0, 0, ARGV_ARRAY_INIT };\n+\n+\tfor (i = 0; i < args->argc; i++) {\n+\t\tconst char *arg = args->argv[i];\n+\t\tconst char *out;\n+\n+\t\tif (!strcmp(\"peel\", arg))\n+\t\t\tdata.peel = 1;\n+\t\telse if (!strcmp(\"symrefs\", arg))\n+\t\t\tdata.symrefs = 1;\n+\t\telse if (skip_prefix(arg, \"ref-pattern \", &out))\n+\t\t\targv_array_pushf(&data.patterns, \"*/%s\", out);\n+\t}\n+\n+\thead_ref_namespaced(send_ref, &data);\n+\tfor_each_namespaced_ref(send_ref, &data);\n+\tpacket_flush(1);\n+\targv_array_clear(&data.patterns);\n+\treturn 0;\n+}\ndiff --git a/ls-refs.h b/ls-refs.h\nnew file mode 100644\nindex 000000000..9e4c57bfe\n--- /dev/null\n+++ b/ls-refs.h\n@@ -0,0 +1,9 @@\n+#ifndef LS_REFS_H\n+#define LS_REFS_H\n+\n+struct repository;\n+struct argv_array;\n+extern int ls_refs(struct repository *r, struct argv_array *keys,\n+\t\t struct argv_array *args);\n+\n+#endif /* LS_REFS_H */\ndiff --git a/serve.c b/serve.c\nindex cf23179b9..c7925c5c7 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -4,8 +4,15 @@\n #include \"pkt-line.h\"\n #include \"version.h\"\n #include \"argv-array.h\"\n+#include \"ls-refs.h\"\n #include \"serve.h\"\n \n+static int always_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\treturn 1;\n+}\n+\n static int agent_advertise(struct repository *r,\n \t\t\t struct strbuf *value)\n {\n@@ -44,6 +51,7 @@ struct protocol_capability {\n \n static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n+\t{ \"ls-refs\", always_advertise, ls_refs },\n };\n \n static void advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex affbad097..33536254e 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -8,6 +8,7 @@ test_expect_success 'test capability advertisement' '\n \tcat >expect <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n+\tls-refs\n \t0000\n \tEOF\n \n@@ -57,4 +58,118 @@ test_expect_success 'request invalid command' '\n \ttest_i18ngrep \"invalid command\" err\n '\n \n+# Test the basics of ls-refs\n+#\n+test_expect_success 'setup some refs and tags' '\n+\ttest_commit one &&\n+\tgit branch dev master &&\n+\ttest_commit two &&\n+\tgit symbolic-ref refs/heads/release refs/heads/master &&\n+\tgit tag -a -m \"annotated tag\" annotated-tag\n+'\n+\n+test_expect_success 'basics of ls-refs' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse HEAD) HEAD\n+\t$(git rev-parse refs/heads/dev) refs/heads/dev\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/heads/release) refs/heads/release\n+\t$(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag\n+\t$(git rev-parse refs/tags/one) refs/tags/one\n+\t$(git rev-parse refs/tags/two) refs/tags/two\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'basic ref-patterns' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tref-pattern master\n+\tref-pattern one\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/tags/one) refs/tags/one\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'wildcard ref-patterns' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tref-pattern refs/heads/*\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/heads/dev) refs/heads/dev\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/heads/release) refs/heads/release\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'peel parameter' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tpeel\n+\tref-pattern refs/tags/*\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag peeled:$(git rev-parse refs/tags/annotated-tag^{})\n+\t$(git rev-parse refs/tags/one) refs/tags/one\n+\t$(git rev-parse refs/tags/two) refs/tags/two\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'symrefs parameter' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tsymrefs\n+\tref-pattern refs/heads/*\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/heads/dev) refs/heads/dev\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/heads/release) refs/heads/release symref-target:refs/heads/master\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n test_done\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338565","messageId":"20180207011312.189834-18-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 17/35] ls-remote: pass ref patterns when requesting a remote's refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:54Z","receivedAt":"2018-02-07T01:14:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct an argv_array of the ref patterns supplied via the command\nline and pass them to 'transport_get_remote_refs()' to be used when\ncommunicating protocol v2 so that the server can limit the ref\nadvertisement based on the supplied patterns.\n\nSigned-off-by: Brandon Williams \n---\n builtin/ls-remote.c | 7 +++++--\n t/t5702-protocol-v2.sh | 16 ++++++++++++++++\n 2 files changed, 21 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex c6e9847c5..caf1051f3 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -43,6 +43,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tint show_symref_target = 0;\n \tconst char *uploadpack = NULL;\n \tconst char **pattern = NULL;\n+\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n \n \tstruct remote *remote;\n \tstruct transport *transport;\n@@ -74,8 +75,10 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (argc > 1) {\n \t\tint i;\n \t\tpattern = xcalloc(argc, sizeof(const char *));\n-\t\tfor (i = 1; i < argc; i++)\n+\t\tfor (i = 1; i < argc; i++) {\n \t\t\tpattern[i - 1] = xstrfmt(\"*/%s\", argv[i]);\n+\t\t\targv_array_push(&ref_patterns, argv[i]);\n+\t\t}\n \t}\n \n \tremote = remote_get(dest);\n@@ -96,7 +99,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (uploadpack != NULL)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n \n-\tref = transport_get_remote_refs(transport, NULL);\n+\tref = transport_get_remote_refs(transport, &ref_patterns);\n \tif (transport_disconnect(transport))\n \t\treturn 1;\n \ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 1e42b5588..a33ff6597 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -30,6 +30,14 @@ test_expect_success 'list refs with git:// using protocol v2' '\n \ttest_cmp actual expect\n '\n \n+test_expect_success 'ref advertisment is filtered with ls-remote using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tls-remote \"$GIT_DAEMON_URL/parent\" master 2>log &&\n+\n+\tgrep \"ref-pattern master\" log &&\n+\t! grep \"refs/tags/\" log\n+'\n+\n stop_git_daemon\n \n # Test protocol v2 with 'file://' transport\n@@ -50,4 +58,12 @@ test_expect_success 'list refs with file:// using protocol v2' '\n \ttest_cmp actual expect\n '\n \n+test_expect_success 'ref advertisment is filtered with ls-remote using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tls-remote \"file://$(pwd)/file_parent\" master 2>log &&\n+\n+\tgrep \"ref-pattern master\" log &&\n+\t! grep \"refs/tags/\" log\n+'\n+\n test_done\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338566","messageId":"20180207011312.189834-19-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 18/35] fetch: pass ref patterns when fetching","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:55Z","receivedAt":"2018-02-07T01:14:03Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct a list of ref patterns to be passed to\n'transport_get_remote_refs()' from the refspec to be used during the\nfetch. This list of ref patterns will be used to allow the server to\nfilter the ref advertisement when communicating using protocol v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch.c | 12 +++++++++++-\n 1 file changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 850382f55..8128450bf 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -332,11 +332,21 @@ static struct ref *get_ref_map(struct transport *transport,\n \tstruct ref *rm;\n \tstruct ref *ref_map = NULL;\n \tstruct ref **tail = &ref_map;\n+\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n \n \t/* opportunistically-updated references: */\n \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n \n-\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n+\tconst struct ref *remote_refs;\n+\n+\tfor (i = 0; i < refspec_count; i++) {\n+\t\tif (!refspecs[i].exact_sha1)\n+\t\t\targv_array_push(&ref_patterns, refspecs[i].src);\n+\t}\n+\n+\tremote_refs = transport_get_remote_refs(transport, &ref_patterns);\n+\n+\targv_array_clear(&ref_patterns);\n \n \tif (refspec_count) {\n \t\tstruct refspec *fetch_refspec;\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338567","messageId":"20180207011312.189834-15-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 14/35] connect: request remote refs using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:51Z","receivedAt":"2018-02-07T01:14:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach the client to be able to request a remote's refs using protocol\nv2. This is done by having a client issue a 'ls-refs' request to a v2\nserver.\n\nSigned-off-by: Brandon Williams \n---\n builtin/upload-pack.c | 10 ++--\n connect.c | 123 ++++++++++++++++++++++++++++++++++++++++++++++++-\n connect.h | 2 +\n remote.h | 4 ++\n t/t5702-protocol-v2.sh | 53 +++++++++++++++++++++\n transport.c | 2 +-\n 6 files changed, 187 insertions(+), 7 deletions(-)\n create mode 100755 t/t5702-protocol-v2.sh\n\ndiff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\nindex 8d53e9794..a757df8da 100644\n--- a/builtin/upload-pack.c\n+++ b/builtin/upload-pack.c\n@@ -5,6 +5,7 @@\n #include \"parse-options.h\"\n #include \"protocol.h\"\n #include \"upload-pack.h\"\n+#include \"serve.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -16,6 +17,7 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \tconst char *dir;\n \tint strict = 0;\n \tstruct upload_pack_options opts = { 0 };\n+\tstruct serve_options serve_opts = SERVE_OPTIONS_INIT;\n \tstruct option options[] = {\n \t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n \t\t\t N_(\"quit after a single request/response exchange\")),\n@@ -48,11 +50,9 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \n \tswitch (determine_protocol_version_server()) {\n \tcase protocol_v2:\n-\t\t/*\n-\t\t * fetch support for protocol v2 has not been implemented yet,\n-\t\t * so ignore the request to use v2 and fallback to using v0.\n-\t\t */\n-\t\tupload_pack(&opts);\n+\t\tserve_opts.advertise_capabilities = opts.advertise_refs;\n+\t\tserve_opts.stateless_rpc = opts.stateless_rpc;\n+\t\tserve(&serve_opts);\n \t\tbreak;\n \tcase protocol_v1:\n \t\t/*\ndiff --git a/connect.c b/connect.c\nindex f2157a821..7cb1f1df7 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -12,9 +12,11 @@\n #include \"sha1-array.h\"\n #include \"transport.h\"\n #include \"strbuf.h\"\n+#include \"version.h\"\n #include \"protocol.h\"\n \n static char *server_capabilities;\n+static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n static const char *parse_feature_value(const char *, const char *, int *);\n \n static int check_ref(const char *name, unsigned int flags)\n@@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n+/* Checks if the server supports the capability 'c' */\n+int server_supports_v2(const char *c, int die_on_error)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n+\t\t (!*out || *out == '='))\n+\t\t\treturn 1;\n+\t}\n+\n+\tif (die_on_error)\n+\t\tdie(\"server doesn't support '%s'\", c);\n+\n+\treturn 0;\n+}\n+\n+static void process_capabilities_v2(struct packet_reader *reader)\n+{\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL)\n+\t\targv_array_push(&server_capabilities_v2, reader->line);\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\tdie(\"protocol error\");\n+}\n+\n enum protocol_version discover_version(struct packet_reader *reader)\n {\n \tenum protocol_version version = protocol_unknown_version;\n@@ -85,7 +114,7 @@ enum protocol_version discover_version(struct packet_reader *reader)\n \t/* Maybe process capabilities here, at least for v2 */\n \tswitch (version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tprocess_capabilities_v2(reader);\n \t\tbreak;\n \tcase protocol_v1:\n \t\t/* Read the peeked version line */\n@@ -293,6 +322,98 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n \treturn list;\n }\n \n+static int process_ref_v2(const char *line, struct ref ***list)\n+{\n+\tint ret = 1;\n+\tint i = 0;\n+\tstruct object_id old_oid;\n+\tstruct ref *ref;\n+\tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n+\n+\tif (string_list_split(&line_sections, line, ' ', -1) < 2) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tif (get_oid_hex(line_sections.items[i++].string, &old_oid)) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tref = alloc_ref(line_sections.items[i++].string);\n+\n+\toidcpy(&ref->old_oid, &old_oid);\n+\t**list = ref;\n+\t*list = &ref->next;\n+\n+\tfor (; i < line_sections.nr; i++) {\n+\t\tconst char *arg = line_sections.items[i].string;\n+\t\tif (skip_prefix(arg, \"symref-target:\", &arg))\n+\t\t\tref->symref = xstrdup(arg);\n+\n+\t\tif (skip_prefix(arg, \"peeled:\", &arg)) {\n+\t\t\tstruct object_id peeled_oid;\n+\t\t\tchar *peeled_name;\n+\t\t\tstruct ref *peeled;\n+\t\t\tif (get_oid_hex(arg, &peeled_oid)) {\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tpeeled_name = xstrfmt(\"%s^{}\", ref->name);\n+\t\t\tpeeled = alloc_ref(peeled_name);\n+\n+\t\t\toidcpy(&peeled->old_oid, &peeled_oid);\n+\t\t\t**list = peeled;\n+\t\t\t*list = &peeled->next;\n+\n+\t\t\tfree(peeled_name);\n+\t\t}\n+\t}\n+\n+out:\n+\tstring_list_clear(&line_sections, 0);\n+\treturn ret;\n+}\n+\n+struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n+\t\t\t struct ref **list, int for_push,\n+\t\t\t const struct argv_array *ref_patterns)\n+{\n+\tint i;\n+\t*list = NULL;\n+\n+\t/* Check that the server supports the ls-refs command */\n+\t/* Issue request for ls-refs */\n+\tif (server_supports_v2(\"ls-refs\", 1))\n+\t\tpacket_write_fmt(fd_out, \"command=ls-refs\\n\");\n+\n+\tif (server_supports_v2(\"agent\", 0))\n+\t packet_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n+\n+\tpacket_delim(fd_out);\n+\t/* When pushing we don't want to request the peeled tags */\n+\tif (!for_push)\n+\t\tpacket_write_fmt(fd_out, \"peel\\n\");\n+\tpacket_write_fmt(fd_out, \"symrefs\\n\");\n+\tfor (i = 0; ref_patterns && i < ref_patterns->argc; i++) {\n+\t\tpacket_write_fmt(fd_out, \"ref-pattern %s\\n\",\n+\t\t\t\t ref_patterns->argv[i]);\n+\t}\n+\tpacket_flush(fd_out);\n+\n+\t/* Process response from server */\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tif (!process_ref_v2(reader->line, &list))\n+\t\t\tdie(\"invalid ls-refs response: %s\", reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\tdie(\"protocol error\");\n+\n+\treturn list;\n+}\n+\n static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)\n {\n \tint len;\ndiff --git a/connect.h b/connect.h\nindex cdb8979dc..8898d4495 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -16,4 +16,6 @@ extern int url_is_local_not_ssh(const char *url);\n struct packet_reader;\n extern enum protocol_version discover_version(struct packet_reader *reader);\n \n+extern int server_supports_v2(const char *c, int die_on_error);\n+\n #endif\ndiff --git a/remote.h b/remote.h\nindex 2016461df..21d0c776c 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -151,10 +151,14 @@ void free_refs(struct ref *ref);\n \n struct oid_array;\n struct packet_reader;\n+struct argv_array;\n extern struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t\t struct ref **list, unsigned int flags,\n \t\t\t\t struct oid_array *extra_have,\n \t\t\t\t struct oid_array *shallow_points);\n+extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n+\t\t\t\t struct ref **list, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nnew file mode 100755\nindex 000000000..1e42b5588\n--- /dev/null\n+++ b/t/t5702-protocol-v2.sh\n@@ -0,0 +1,53 @@\n+#!/bin/sh\n+\n+test_description='test git wire-protocol version 2'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\ttest_commit -C \"$daemon_parent\" one\n+'\n+\n+test_expect_success 'list refs with git:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tls-remote --symref \"$GIT_DAEMON_URL/parent\" >actual 2>log &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"git> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log &&\n+\n+\tgit ls-remote --symref \"$GIT_DAEMON_URL/parent\" >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+stop_git_daemon\n+\n+# Test protocol v2 with 'file://' transport\n+#\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init file_parent &&\n+\ttest_commit -C file_parent one\n+'\n+\n+test_expect_success 'list refs with file:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tls-remote --symref \"file://$(pwd)/file_parent\" >actual 2>log &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log &&\n+\n+\tgit ls-remote --symref \"file://$(pwd)/file_parent\" >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/transport.c b/transport.c\nindex 83d9dd1df..ffc6b2614 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -204,7 +204,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338568","messageId":"20180207011312.189834-17-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 16/35] transport: convert transport_get_remote_refs to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:53Z","receivedAt":"2018-02-07T01:14:08Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert 'transport_get_remote_refs()' to optionally take a list of ref\npatterns.\n\nSigned-off-by: Brandon Williams \n---\n builtin/clone.c | 2 +-\n builtin/fetch.c | 4 ++--\n builtin/ls-remote.c | 2 +-\n builtin/remote.c | 2 +-\n transport.c | 7 +++++--\n transport.h | 3 ++-\n 6 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 284651797..6e77d993f 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1121,7 +1121,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (transport->smart_options && !deepen)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n-\trefs = transport_get_remote_refs(transport);\n+\trefs = transport_get_remote_refs(transport, NULL);\n \n \tif (refs) {\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7bbcd26fa..850382f55 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -250,7 +250,7 @@ static void find_non_local_tags(struct transport *transport,\n \tstruct string_list_item *item = NULL;\n \n \tfor_each_ref(add_existing, &existing_refs);\n-\tfor (ref = transport_get_remote_refs(transport); ref; ref = ref->next) {\n+\tfor (ref = transport_get_remote_refs(transport, NULL); ref; ref = ref->next) {\n \t\tif (!starts_with(ref->name, \"refs/tags/\"))\n \t\t\tcontinue;\n \n@@ -336,7 +336,7 @@ static struct ref *get_ref_map(struct transport *transport,\n \t/* opportunistically-updated references: */\n \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n \n-\tconst struct ref *remote_refs = transport_get_remote_refs(transport);\n+\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n \n \tif (refspec_count) {\n \t\tstruct refspec *fetch_refspec;\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex c4be98ab9..c6e9847c5 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -96,7 +96,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (uploadpack != NULL)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n \n-\tref = transport_get_remote_refs(transport);\n+\tref = transport_get_remote_refs(transport, NULL);\n \tif (transport_disconnect(transport))\n \t\treturn 1;\n \ndiff --git a/builtin/remote.c b/builtin/remote.c\nindex d95bf904c..d0b6ff6e2 100644\n--- a/builtin/remote.c\n+++ b/builtin/remote.c\n@@ -862,7 +862,7 @@ static int get_remote_ref_states(const char *name,\n \tif (query) {\n \t\ttransport = transport_get(states->remote, states->remote->url_nr > 0 ?\n \t\t\tstates->remote->url[0] : NULL);\n-\t\tremote_refs = transport_get_remote_refs(transport);\n+\t\tremote_refs = transport_get_remote_refs(transport, NULL);\n \t\ttransport_disconnect(transport);\n \n \t\tstates->queried = 1;\ndiff --git a/transport.c b/transport.c\nindex c54a44630..dfc603b36 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1136,10 +1136,13 @@ int transport_push(struct transport *transport,\n \treturn 1;\n }\n \n-const struct ref *transport_get_remote_refs(struct transport *transport)\n+const struct ref *transport_get_remote_refs(struct transport *transport,\n+\t\t\t\t\t const struct argv_array *ref_patterns)\n {\n \tif (!transport->got_remote_refs) {\n-\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n+\t\ttransport->remote_refs =\n+\t\t\ttransport->vtable->get_refs_list(transport, 0,\n+\t\t\t\t\t\t\t ref_patterns);\n \t\ttransport->got_remote_refs = 1;\n \t}\n \ndiff --git a/transport.h b/transport.h\nindex 731c78b67..4b656f315 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -178,7 +178,8 @@ int transport_push(struct transport *connection,\n \t\t int refspec_nr, const char **refspec, int flags,\n \t\t unsigned int * reject_reasons);\n \n-const struct ref *transport_get_remote_refs(struct transport *transport);\n+const struct ref *transport_get_remote_refs(struct transport *transport,\n+\t\t\t\t\t const struct argv_array *ref_patterns);\n \n int transport_fetch_refs(struct transport *transport, struct ref *refs);\n void transport_unlock_pack(struct transport *transport);\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338569","messageId":"20180207011312.189834-22-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 21/35] fetch-pack: perform a fetch using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:58Z","receivedAt":"2018-02-07T01:14:11Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"When communicating with a v2 server, perform a fetch by requesting the\n'fetch' command.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 2 +-\n fetch-pack.c | 252 ++++++++++++++++++++++++++++++++++++++++++++++++-\n fetch-pack.h | 4 +-\n t/t5702-protocol-v2.sh | 84 +++++++++++++++++\n transport.c | 7 +-\n 5 files changed, 342 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex f492e8abd..867dd3cc7 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -213,7 +213,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n-\t\t\t &shallow, pack_lockfile_ptr);\n+\t\t\t &shallow, pack_lockfile_ptr, protocol_v0);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 9f6b07ad9..4fb5805dd 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1008,6 +1008,247 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \treturn ref;\n }\n \n+static void add_wants(const struct ref *wants, struct strbuf *req_buf)\n+{\n+\tfor ( ; wants ; wants = wants->next) {\n+\t\tconst struct object_id *remote = &wants->old_oid;\n+\t\tconst char *remote_hex;\n+\t\tstruct object *o;\n+\n+\t\t/*\n+\t\t * If that object is complete (i.e. it is an ancestor of a\n+\t\t * local ref), we tell them we have it but do not have to\n+\t\t * tell them about its ancestors, which they already know\n+\t\t * about.\n+\t\t *\n+\t\t * We use lookup_object here because we are only\n+\t\t * interested in the case we *know* the object is\n+\t\t * reachable and we have already scanned it.\n+\t\t */\n+\t\tif (((o = lookup_object(remote->hash)) != NULL) &&\n+\t\t (o->flags & COMPLETE)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tremote_hex = oid_to_hex(remote);\n+\t\tpacket_buf_write(req_buf, \"want %s\\n\", remote_hex);\n+\t}\n+}\n+\n+static void add_common(struct strbuf *req_buf, struct oidset *common)\n+{\n+\tstruct oidset_iter iter;\n+\tconst struct object_id *oid;\n+\toidset_iter_init(common, &iter);\n+\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n+\t}\n+}\n+\n+static int add_haves(struct strbuf *req_buf, int *in_vain)\n+{\n+\tint ret = 0;\n+\tint haves_added = 0;\n+\tconst struct object_id *oid;\n+\n+\twhile ((oid = get_rev())) {\n+\t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n+\t\tif (++haves_added >= INITIAL_FLUSH)\n+\t\t\tbreak;\n+\t};\n+\n+\t*in_vain += haves_added;\n+\tif (!haves_added || *in_vain >= MAX_IN_VAIN) {\n+\t\t/* Send Done */\n+\t\tpacket_buf_write(req_buf, \"done\\n\");\n+\t\tret = 1;\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n+\t\t\t const struct ref *wants, struct oidset *common,\n+\t\t\t int *in_vain)\n+{\n+\tint ret = 0;\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\n+\tif (server_supports_v2(\"fetch\", 1))\n+\t\tpacket_buf_write(&req_buf, \"command=fetch\");\n+\tif (server_supports_v2(\"agent\", 0))\n+\t\tpacket_buf_write(&req_buf, \"agent=%s\", git_user_agent_sanitized());\n+\n+\tpacket_buf_delim(&req_buf);\n+\tif (args->use_thin_pack)\n+\t\tpacket_buf_write(&req_buf, \"thin-pack\");\n+\tif (args->no_progress)\n+\t\tpacket_buf_write(&req_buf, \"no-progress\");\n+\tif (args->include_tag)\n+\t\tpacket_buf_write(&req_buf, \"include-tag\");\n+\tif (prefer_ofs_delta)\n+\t\tpacket_buf_write(&req_buf, \"ofs-delta\");\n+\n+\t/* add wants */\n+\tadd_wants(wants, &req_buf);\n+\n+\t/* Add all of the common commits we've found in previous rounds */\n+\tadd_common(&req_buf, common);\n+\n+\t/* Add initial haves */\n+\tret = add_haves(&req_buf, in_vain);\n+\n+\t/* Send request */\n+\tpacket_buf_flush(&req_buf);\n+\twrite_or_die(fd_out, req_buf.buf, req_buf.len);\n+\n+\tstrbuf_release(&req_buf);\n+\treturn ret;\n+}\n+\n+/*\n+ * Processes a section header in a server's response and checks if it matches\n+ * `section`. If the value of `peek` is 1, the header line will be peeked (and\n+ * not consumed); if 0, the line will be consumed and the function will die if\n+ * the section header doesn't match what was expected.\n+ */\n+static int process_section_header(struct packet_reader *reader,\n+\t\t\t\t const char *section, int peek)\n+{\n+\tint ret;\n+\n+\tif (packet_reader_peek(reader) != PACKET_READ_NORMAL)\n+\t\tdie(\"error reading packet\");\n+\n+\tret = !strcmp(reader->line, section);\n+\n+\tif (!peek) {\n+\t\tif (!ret)\n+\t\t\tdie(\"expected '%s', received '%s'\",\n+\t\t\t section, reader->line);\n+\t\tpacket_reader_read(reader);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static int process_acks(struct packet_reader *reader, struct oidset *common)\n+{\n+\t/* received */\n+\tint received_ready = 0;\n+\tint received_ack = 0;\n+\n+\tprocess_section_header(reader, \"acknowledgments\", 0);\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *arg;\n+\n+\t\tif (!strcmp(reader->line, \"NAK\"))\n+\t\t\tcontinue;\n+\n+\t\tif (skip_prefix(reader->line, \"ACK \", &arg)) {\n+\t\t\tstruct object_id oid;\n+\t\t\tif (!get_oid_hex(arg, &oid)) {\n+\t\t\t\tstruct commit *commit;\n+\t\t\t\toidset_insert(common, &oid);\n+\t\t\t\tcommit = lookup_commit(&oid);\n+\t\t\t\tmark_common(commit, 0, 1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!strcmp(reader->line, \"ready\")) {\n+\t\t\tclear_prio_queue(&rev_list);\n+\t\t\treceived_ready = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tdie(_(\"git fetch-pack: expected ACK/NAK, got '%s'\"), reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH &&\n+\t reader->status != PACKET_READ_DELIM)\n+\t\tdie(\"Error during processing acks: %d\", reader->status);\n+\n+\t/* return 0 if no common, 1 if there are common, or 2 if ready */\n+\treturn received_ready ? 2 : (received_ack ? 1 : 0);\n+}\n+\n+enum fetch_state {\n+\tFETCH_CHECK_LOCAL = 0,\n+\tFETCH_SEND_REQUEST,\n+\tFETCH_PROCESS_ACKS,\n+\tFETCH_GET_PACK,\n+\tFETCH_DONE,\n+};\n+\n+static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n+\t\t\t\t int fd[2],\n+\t\t\t\t const struct ref *orig_ref,\n+\t\t\t\t struct ref **sought, int nr_sought,\n+\t\t\t\t char **pack_lockfile)\n+{\n+\tstruct ref *ref = copy_ref_list(orig_ref);\n+\tenum fetch_state state = FETCH_CHECK_LOCAL;\n+\tstruct oidset common = OIDSET_INIT;\n+\tstruct packet_reader reader;\n+\tint in_vain = 0;\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (state != FETCH_DONE) {\n+\t\tswitch (state) {\n+\t\tcase FETCH_CHECK_LOCAL:\n+\t\t\tsort_ref_list(&ref, ref_compare_name);\n+\t\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n+\n+\t\t\t/* v2 supports these by default */\n+\t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n+\t\t\tuse_sideband = 2;\n+\n+\t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n+\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_SEND_REQUEST;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_REQUEST:\n+\t\t\tif (send_fetch_request(fd[1], args, ref, &common, &in_vain))\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_PROCESS_ACKS;\n+\t\t\tbreak;\n+\t\tcase FETCH_PROCESS_ACKS:\n+\t\t\t/* Process ACKs/NAKs */\n+\t\t\tswitch (process_acks(&reader, &common)) {\n+\t\t\tcase 2:\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\t\tbreak;\n+\t\t\tcase 1:\n+\t\t\t\tin_vain = 0;\n+\t\t\t\t/* fallthrough */\n+\t\t\tdefault:\n+\t\t\t\tstate = FETCH_SEND_REQUEST;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase FETCH_GET_PACK:\n+\t\t\t/* get the pack */\n+\t\t\tprocess_section_header(&reader, \"packfile\", 0);\n+\t\t\tif (get_pack(args, fd, pack_lockfile))\n+\t\t\t\tdie(_(\"git fetch-pack: fetch failed.\"));\n+\n+\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_DONE:\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n+\toidset_clear(&common);\n+\treturn ref;\n+}\n+\n static void fetch_pack_config(void)\n {\n \tgit_config_get_int(\"fetch.unpacklimit\", &fetch_unpack_limit);\n@@ -1153,7 +1394,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const char *dest,\n \t\t struct ref **sought, int nr_sought,\n \t\t struct oid_array *shallow,\n-\t\t char **pack_lockfile)\n+\t\t char **pack_lockfile,\n+\t\t enum protocol_version version)\n {\n \tstruct ref *ref_cpy;\n \tstruct shallow_info si;\n@@ -1167,8 +1409,12 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tdie(_(\"no matching remote head\"));\n \t}\n \tprepare_shallow_info(&si, shallow);\n-\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n-\t\t\t\t&si, pack_lockfile);\n+\tif (version == protocol_v2)\n+\t\tref_cpy = do_fetch_pack_v2(args, fd, ref, sought, nr_sought,\n+\t\t\t\t\t pack_lockfile);\n+\telse\n+\t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n+\t\t\t\t\t&si, pack_lockfile);\n \treprepare_packed_git();\n \tupdate_shallow(args, sought, nr_sought, &si);\n \tclear_shallow_info(&si);\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex b6aeb43a8..7afca7305 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -3,6 +3,7 @@\n \n #include \"string-list.h\"\n #include \"run-command.h\"\n+#include \"protocol.h\"\n \n struct oid_array;\n \n@@ -43,7 +44,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t struct ref **sought,\n \t\t int nr_sought,\n \t\t struct oid_array *shallow,\n-\t\t char **pack_lockfile);\n+\t\t char **pack_lockfile,\n+\t\t enum protocol_version version);\n \n /*\n * Print an appropriate error message for each sought ref that wasn't\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex a33ff6597..16304d1c8 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -38,6 +38,50 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol\n \t! grep \"refs/tags/\" log\n '\n \n+test_expect_success 'clone with git:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tclone \"$GIT_DAEMON_URL/parent\" daemon_child 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"clone> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"clone< version 2\" log\n+'\n+\n+test_expect_success 'fetch with git:// using protocol v2' '\n+\ttest_commit -C \"$daemon_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=2 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"fetch< version 2\" log\n+'\n+\n+test_expect_success 'pull with git:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=2 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"fetch< version 2\" log\n+'\n+\n stop_git_daemon\n \n # Test protocol v2 with 'file://' transport\n@@ -66,4 +110,44 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol\n \t! grep \"refs/tags/\" log\n '\n \n+test_expect_success 'clone with file:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n+\t\tclone \"file://$(pwd)/file_parent\" file_child 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"clone< version 2\" log\n+'\n+\n+test_expect_success 'fetch with file:// using protocol v2' '\n+\ttest_commit -C file_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n+\t\tfetch origin 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"fetch< version 2\" log\n+'\n+\n+test_expect_success 'ref advertisment is filtered during fetch using protocol v2' '\n+\ttest_commit -C file_parent three &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n+\t\tfetch origin master 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\tgrep \"ref-pattern master\" log &&\n+\t! grep \"refs/tags/\" log\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex 6ea3905e3..c275f46ed 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -256,14 +256,17 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\trefs = fetch_pack(&args, data->fd, data->conn,\n+\t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n+\t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n+\t\t\t\t &transport->pack_lockfile, data->version);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n \t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n-\t\t\t\t &transport->pack_lockfile);\n+\t\t\t\t &transport->pack_lockfile, data->version);\n \t\tbreak;\n \tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338570","messageId":"20180207011312.189834-23-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 22/35] upload-pack: support shallow requests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:59Z","receivedAt":"2018-02-07T01:14:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add the 'shallow' feature to the protocol version 2 command 'fetch'\nwhich indicates that the server supports shallow clients and deepen\nrequets.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/technical/protocol-v2.txt | 67 +++++++++++++++-\n serve.c | 2 +-\n t/t5701-git-serve.sh | 2 +-\n upload-pack.c | 138 +++++++++++++++++++++++---------\n upload-pack.h | 3 +\n 5 files changed, 173 insertions(+), 39 deletions(-)\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 4d5096dae..fedeb6b77 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -201,12 +201,42 @@ packet-lines:\n \tto its base by position in pack rather than by an oid. That is,\n \tthey can read OBJ_OFS_DELTA (ake type 6) in a packfile.\n \n+ shallow \n+\tA client must notify the server of all objects for which it only\n+\thas shallow copies of (meaning that it doesn't have the parents\n+\tof a commit) by supplying a 'shallow ' line for each such\n+\tobject so that the serve is aware of the limitations of the\n+\tclient's history.\n+\n+ deepen \n+\tRequest that the fetch/clone should be shallow having a commit depth of\n+\t relative to the remote side.\n+\n+ deepen-relative\n+\tRequests that the semantics of the \"deepen\" command be changed\n+\tto indicate that the depth requested is relative to the clients\n+\tcurrent shallow boundary, instead of relative to the remote\n+\trefs.\n+\n+ deepen-since \n+\tRequests that the shallow clone/fetch should be cut at a\n+\tspecific time, instead of depth. Internally it's equivalent of\n+\tdoing \"rev-list --max-age=\". Cannot be used with\n+\t\"deepen\".\n+\n+ deepen-not \n+\tRequests that the shallow clone/fetch should be cut at a\n+\tspecific revision specified by '', instead of a depth.\n+\tInternally it's equivalent of doing \"rev-list --not \".\n+\tCannot be used with \"deepen\", but can be used with\n+\t\"deepen-since\".\n+\n The response of `fetch` is broken into a number of sections separated by\n delimiter packets (0001), with each section beginning with its section\n header.\n \n output = *section\n- section = (acknowledgments | packfile)\n+ section = (acknowledgments | shallow-info | packfile)\n \t (flush-pkt | delim-pkt)\n \n acknowledgments = PKT-LINE(\"acknowledgments\" LF)\n@@ -215,6 +245,11 @@ header.\n nak = PKT-LINE(\"NAK\" LF)\n ack = PKT-LINE(\"ACK\" SP obj-id LF)\n \n+ shallow-info = PKT-LINE(\"shallow-info\" LF)\n+\t\t *PKT-LINE((shallow | unshallow) LF)\n+ shallow = \"shallow\" SP obj-id\n+ unshallow = \"unshallow\" SP obj-id\n+\n packfile = PKT-LINE(\"packfile\" LF)\n \t [PACKFILE]\n \n@@ -247,6 +282,36 @@ header.\n \t determined the objects it plans to send to the client and no\n \t further negotiation is needed.\n \n+----\n+ shallow-info section\n+\tIf the client has requested a shallow fetch/clone, a shallow\n+\tclient requests a fetch or the server is shallow then the\n+\tserver's response may include a shallow-info section. The\n+\tshallow-info section will be include if (due to one of the above\n+\tconditions) the server needs to inform the client of any shallow\n+\tboundaries or adjustments to the clients already existing\n+\tshallow boundaries.\n+\n+\t* Always begins with the section header \"shallow-info\"\n+\n+\t* If a positive depth is requested, the server will compute the\n+\t set of commits which are no deeper than the desired depth.\n+\n+\t* The server sends a \"shallow obj-id\" line for each commit whose\n+\t parents will not be sent in the following packfile.\n+\n+\t* The server sends an \"unshallow obj-id\" line for each commit\n+\t which the client has indicated is shallow, but is no longer\n+\t shallow as a result of the fetch (due to its parents being\n+\t sent in the following packfile).\n+\n+\t* The server MUST NOT send any \"unshallow\" lines for anything\n+\t which the client has not indicated was shallow as a part of\n+\t its request.\n+\n+\t* This section is only included if a packfile section is also\n+\t included in the response.\n+\n ----\n packfile section\n \t* Always begins with the section header \"packfile\"\ndiff --git a/serve.c b/serve.c\nindex 05cc434cf..c3e58c1e7 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -53,7 +53,7 @@ struct protocol_capability {\n static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n \t{ \"ls-refs\", always_advertise, ls_refs },\n-\t{ \"fetch\", always_advertise, upload_pack_v2 },\n+\t{ \"fetch\", upload_pack_advertise, upload_pack_v2 },\n };\n \n static void advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 202cb782d..491adc693 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -9,7 +9,7 @@ test_expect_success 'test capability advertisement' '\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs\n-\tfetch\n+\tfetch=shallow\n \t0000\n \tEOF\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex c6518a24d..a7e4f9e9c 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -710,7 +710,6 @@ static void deepen(int depth, int deepen_relative,\n \t}\n \n \tsend_unshallow(shallows);\n-\tpacket_flush(1);\n }\n \n static void deepen_by_rev_list(int ac, const char **av,\n@@ -722,7 +721,52 @@ static void deepen_by_rev_list(int ac, const char **av,\n \tsend_shallow(result);\n \tfree_commit_list(result);\n \tsend_unshallow(shallows);\n-\tpacket_flush(1);\n+}\n+\n+static int send_shallow_list(int depth, int deepen_rev_list,\n+\t\t\t timestamp_t deepen_since,\n+\t\t\t struct string_list *deepen_not,\n+\t\t\t struct object_array *shallows)\n+{\n+\tint ret = 0;\n+\n+\tif (depth > 0 && deepen_rev_list)\n+\t\tdie(\"git upload-pack: deepen and deepen-since (or deepen-not) cannot be used together\");\n+\tif (depth > 0) {\n+\t\tdeepen(depth, deepen_relative, shallows);\n+\t\tret = 1;\n+\t} else if (deepen_rev_list) {\n+\t\tstruct argv_array av = ARGV_ARRAY_INIT;\n+\t\tint i;\n+\n+\t\targv_array_push(&av, \"rev-list\");\n+\t\tif (deepen_since)\n+\t\t\targv_array_pushf(&av, \"--max-age=%\"PRItime, deepen_since);\n+\t\tif (deepen_not->nr) {\n+\t\t\targv_array_push(&av, \"--not\");\n+\t\t\tfor (i = 0; i < deepen_not->nr; i++) {\n+\t\t\t\tstruct string_list_item *s = deepen_not->items + i;\n+\t\t\t\targv_array_push(&av, s->string);\n+\t\t\t}\n+\t\t\targv_array_push(&av, \"--not\");\n+\t\t}\n+\t\tfor (i = 0; i < want_obj.nr; i++) {\n+\t\t\tstruct object *o = want_obj.objects[i].item;\n+\t\t\targv_array_push(&av, oid_to_hex(&o->oid));\n+\t\t}\n+\t\tdeepen_by_rev_list(av.argc, av.argv, shallows);\n+\t\targv_array_clear(&av);\n+\t\tret = 1;\n+\t} else {\n+\t\tif (shallows->nr > 0) {\n+\t\t\tint i;\n+\t\t\tfor (i = 0; i < shallows->nr; i++)\n+\t\t\t\tregister_shallow(&shallows->objects[i].item->oid);\n+\t\t}\n+\t}\n+\n+\tshallow_nr += shallows->nr;\n+\treturn ret;\n }\n \n static int process_shallow(const char *line, struct object_array *shallows)\n@@ -884,40 +928,10 @@ static void receive_needs(void)\n \n \tif (depth == 0 && !deepen_rev_list && shallows.nr == 0)\n \t\treturn;\n-\tif (depth > 0 && deepen_rev_list)\n-\t\tdie(\"git upload-pack: deepen and deepen-since (or deepen-not) cannot be used together\");\n-\tif (depth > 0)\n-\t\tdeepen(depth, deepen_relative, &shallows);\n-\telse if (deepen_rev_list) {\n-\t\tstruct argv_array av = ARGV_ARRAY_INIT;\n-\t\tint i;\n \n-\t\targv_array_push(&av, \"rev-list\");\n-\t\tif (deepen_since)\n-\t\t\targv_array_pushf(&av, \"--max-age=%\"PRItime, deepen_since);\n-\t\tif (deepen_not.nr) {\n-\t\t\targv_array_push(&av, \"--not\");\n-\t\t\tfor (i = 0; i < deepen_not.nr; i++) {\n-\t\t\t\tstruct string_list_item *s = deepen_not.items + i;\n-\t\t\t\targv_array_push(&av, s->string);\n-\t\t\t}\n-\t\t\targv_array_push(&av, \"--not\");\n-\t\t}\n-\t\tfor (i = 0; i < want_obj.nr; i++) {\n-\t\t\tstruct object *o = want_obj.objects[i].item;\n-\t\t\targv_array_push(&av, oid_to_hex(&o->oid));\n-\t\t}\n-\t\tdeepen_by_rev_list(av.argc, av.argv, &shallows);\n-\t\targv_array_clear(&av);\n-\t}\n-\telse\n-\t\tif (shallows.nr > 0) {\n-\t\t\tint i;\n-\t\t\tfor (i = 0; i < shallows.nr; i++)\n-\t\t\t\tregister_shallow(&shallows.objects[i].item->oid);\n-\t\t}\n-\n-\tshallow_nr += shallows.nr;\n+\tif (send_shallow_list(depth, deepen_rev_list, deepen_since,\n+\t\t\t &deepen_not, &shallows))\n+\t\tpacket_flush(1);\n \tobject_array_clear(&shallows);\n }\n \n@@ -1071,6 +1085,13 @@ struct upload_pack_data {\n \tstruct object_array wants;\n \tstruct oid_array haves;\n \n+\tstruct object_array shallows;\n+\tstruct string_list deepen_not;\n+\tint depth;\n+\ttimestamp_t deepen_since;\n+\tint deepen_rev_list;\n+\tint deepen_relative;\n+\n \tunsigned stateless_rpc : 1;\n \n \tunsigned use_thin_pack : 1;\n@@ -1080,12 +1101,14 @@ struct upload_pack_data {\n \tunsigned done : 1;\n };\n \n-#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }\n+#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, OBJECT_ARRAY_INIT, STRING_LIST_INIT_DUP, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }\n \n static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tobject_array_clear(&data->wants);\n \toid_array_clear(&data->haves);\n+\tobject_array_clear(&data->shallows);\n+\tstring_list_clear(&data->deepen_not, 0);\n }\n \n static int parse_want(const char *line)\n@@ -1170,6 +1193,22 @@ static void process_args(struct argv_array *args, struct upload_pack_data *data)\n \t\t\tcontinue;\n \t\t}\n \n+\t\t/* Shallow related arguments */\n+\t\tif (process_shallow(arg, &data->shallows))\n+\t\t\tcontinue;\n+\t\tif (process_deepen(arg, &data->depth))\n+\t\t\tcontinue;\n+\t\tif (process_deepen_since(arg, &data->deepen_since,\n+\t\t\t\t\t &data->deepen_rev_list))\n+\t\t\tcontinue;\n+\t\tif (process_deepen_not(arg, &data->deepen_not,\n+\t\t\t\t &data->deepen_rev_list))\n+\t\t\tcontinue;\n+\t\tif (!strcmp(arg, \"deepen-relative\")) {\n+\t\t\tdata->deepen_relative = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\t/* ignore unknown lines maybe? */\n \t\tdie(\"unexpect line: '%s'\", arg);\n \t}\n@@ -1284,6 +1323,23 @@ static int process_haves_and_send_acks(struct upload_pack_data *data)\n \treturn ret;\n }\n \n+static void send_shallow_info(struct upload_pack_data *data)\n+{\n+\t/* No shallow info needs to be sent */\n+\tif (!data->depth && !data->deepen_rev_list && !data->shallows.nr &&\n+\t !is_repository_shallow())\n+\t\treturn;\n+\n+\tpacket_write_fmt(1, \"shallow-info\\n\");\n+\n+\tif (!send_shallow_list(data->depth, data->deepen_rev_list,\n+\t\t\t data->deepen_since, &data->deepen_not,\n+\t\t\t &data->shallows) && is_repository_shallow())\n+\t\tdeepen(INFINITE_DEPTH, data->deepen_relative, &data->shallows);\n+\n+\tpacket_delim(1);\n+}\n+\n enum fetch_state {\n \tFETCH_PROCESS_ARGS = 0,\n \tFETCH_READ_HAVES,\n@@ -1334,6 +1390,8 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys,\n \t\t\t\tstate = FETCH_DONE;\n \t\t\tbreak;\n \t\tcase FETCH_SEND_PACK:\n+\t\t\tsend_shallow_info(&data);\n+\n \t\t\tpacket_write_fmt(1, \"packfile\\n\");\n \t\t\tcreate_pack_file();\n \t\t\tstate = FETCH_DONE;\n@@ -1346,3 +1404,11 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys,\n \tupload_pack_data_clear(&data);\n \treturn 0;\n }\n+\n+int upload_pack_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, \"shallow\");\n+\treturn 1;\n+}\ndiff --git a/upload-pack.h b/upload-pack.h\nindex 6b7890238..7720f2142 100644\n--- a/upload-pack.h\n+++ b/upload-pack.h\n@@ -14,5 +14,8 @@ struct repository;\n struct argv_array;\n extern int upload_pack_v2(struct repository *r, struct argv_array *keys,\n \t\t\t struct argv_array *args);\n+struct strbuf;\n+extern int upload_pack_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value);\n \n #endif /* UPLOAD_PACK_H */\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338571","messageId":"20180207011312.189834-20-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 19/35] push: pass ref patterns when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:56Z","receivedAt":"2018-02-07T01:14:19Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct a list of ref patterns to be passed to 'get_refs_list()' from\nthe refspec to be used during the push. This list of ref patterns will\nbe used to allow the server to filter the ref advertisement when\ncommunicating using protocol v2.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 17 ++++++++++++++++-\n 1 file changed, 16 insertions(+), 1 deletion(-)\n\ndiff --git a/transport.c b/transport.c\nindex dfc603b36..6ea3905e3 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1026,11 +1026,26 @@ int transport_push(struct transport *transport,\n \t\tint porcelain = flags & TRANSPORT_PUSH_PORCELAIN;\n \t\tint pretend = flags & TRANSPORT_PUSH_DRY_RUN;\n \t\tint push_ret, ret, err;\n+\t\tstruct refspec *tmp_rs;\n+\t\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n+\t\tint i;\n \n \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n \t\t\treturn -1;\n \n-\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n+\t\ttmp_rs = parse_push_refspec(refspec_nr, refspec);\n+\t\tfor (i = 0; i < refspec_nr; i++) {\n+\t\t\tif (tmp_rs[i].dst)\n+\t\t\t\targv_array_push(&ref_patterns, tmp_rs[i].dst);\n+\t\t\telse if (tmp_rs[i].src && !tmp_rs[i].exact_sha1)\n+\t\t\t\targv_array_push(&ref_patterns, tmp_rs[i].src);\n+\t\t}\n+\n+\t\tremote_refs = transport->vtable->get_refs_list(transport, 1,\n+\t\t\t\t\t\t\t &ref_patterns);\n+\n+\t\targv_array_clear(&ref_patterns);\n+\t\tfree_refspec(refspec_nr, tmp_rs);\n \n \t\tif (flags & TRANSPORT_PUSH_ALL)\n \t\t\tmatch_flags |= MATCH_REFS_ALL;\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338572","messageId":"20180207011312.189834-24-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 23/35] fetch-pack: support shallow requests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:00Z","receivedAt":"2018-02-07T01:14:23Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Enable shallow clones and deepen requests using protocol version 2 if\nthe server 'fetch' command supports the 'shallow' feature.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 22 +++++++++++++++++++\n connect.h | 2 ++\n fetch-pack.c | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 3 files changed, 92 insertions(+), 1 deletion(-)\n\ndiff --git a/connect.c b/connect.c\nindex 7cb1f1df7..9577528f3 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -82,6 +82,28 @@ int server_supports_v2(const char *c, int die_on_error)\n \treturn 0;\n }\n \n+int server_supports_feature(const char *c, const char *feature,\n+\t\t\t int die_on_error)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n+\t\t (!*out || *(out++) == '=')) {\n+\t\t\tif (parse_feature_request(out, feature))\n+\t\t\t\treturn 1;\n+\t\t\telse\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (die_on_error)\n+\t\tdie(\"server doesn't support feature '%s'\", feature);\n+\n+\treturn 0;\n+}\n+\n static void process_capabilities_v2(struct packet_reader *reader)\n {\n \twhile (packet_reader_read(reader) == PACKET_READ_NORMAL)\ndiff --git a/connect.h b/connect.h\nindex 8898d4495..0e69c6709 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -17,5 +17,7 @@ struct packet_reader;\n extern enum protocol_version discover_version(struct packet_reader *reader);\n \n extern int server_supports_v2(const char *c, int die_on_error);\n+extern int server_supports_feature(const char *c, const char *feature,\n+\t\t\t\t int die_on_error);\n \n #endif\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 4fb5805dd..c0807e219 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1008,6 +1008,26 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \treturn ref;\n }\n \n+static void add_shallow_requests(struct strbuf *req_buf,\n+\t\t\t\t const struct fetch_pack_args *args)\n+{\n+\tif (is_repository_shallow())\n+\t\twrite_shallow_commits(req_buf, 1, NULL);\n+\tif (args->depth > 0)\n+\t\tpacket_buf_write(req_buf, \"deepen %d\", args->depth);\n+\tif (args->deepen_since) {\n+\t\ttimestamp_t max_age = approxidate(args->deepen_since);\n+\t\tpacket_buf_write(req_buf, \"deepen-since %\"PRItime, max_age);\n+\t}\n+\tif (args->deepen_not) {\n+\t\tint i;\n+\t\tfor (i = 0; i < args->deepen_not->nr; i++) {\n+\t\t\tstruct string_list_item *s = args->deepen_not->items + i;\n+\t\t\tpacket_buf_write(req_buf, \"deepen-not %s\", s->string);\n+\t\t}\n+\t}\n+}\n+\n static void add_wants(const struct ref *wants, struct strbuf *req_buf)\n {\n \tfor ( ; wants ; wants = wants->next) {\n@@ -1090,6 +1110,10 @@ static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n \tif (prefer_ofs_delta)\n \t\tpacket_buf_write(&req_buf, \"ofs-delta\");\n \n+\t/* Add shallow-info and deepen request */\n+\tif (server_supports_feature(\"fetch\", \"shallow\", 1))\n+\t\tadd_shallow_requests(&req_buf, args);\n+\n \t/* add wants */\n \tadd_wants(wants, &req_buf);\n \n@@ -1119,7 +1143,7 @@ static int process_section_header(struct packet_reader *reader,\n \tint ret;\n \n \tif (packet_reader_peek(reader) != PACKET_READ_NORMAL)\n-\t\tdie(\"error reading packet\");\n+\t\tdie(\"error reading section header '%s'\", section);\n \n \tret = !strcmp(reader->line, section);\n \n@@ -1174,6 +1198,43 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n \treturn received_ready ? 2 : (received_ack ? 1 : 0);\n }\n \n+static void receive_shallow_info(struct fetch_pack_args *args,\n+\t\t\t\t struct packet_reader *reader)\n+{\n+\tprocess_section_header(reader, \"shallow-info\", 0);\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *arg;\n+\t\tstruct object_id oid;\n+\n+\t\tif (skip_prefix(reader->line, \"shallow \", &arg)) {\n+\t\t\tif (get_oid_hex(arg, &oid))\n+\t\t\t\tdie(_(\"invalid shallow line: %s\"), reader->line);\n+\t\t\tregister_shallow(&oid);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(reader->line, \"unshallow \", &arg)) {\n+\t\t\tif (get_oid_hex(arg, &oid))\n+\t\t\t\tdie(_(\"invalid unshallow line: %s\"), reader->line);\n+\t\t\tif (!lookup_object(oid.hash))\n+\t\t\t\tdie(_(\"object not found: %s\"), reader->line);\n+\t\t\t/* make sure that it is parsed as shallow */\n+\t\t\tif (!parse_object(&oid))\n+\t\t\t\tdie(_(\"error in object: %s\"), reader->line);\n+\t\t\tif (unregister_shallow(&oid))\n+\t\t\t\tdie(_(\"no shallow found: %s\"), reader->line);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tdie(_(\"expected shallow/unshallow, got %s\"), reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH &&\n+\t reader->status != PACKET_READ_DELIM)\n+\t\tdie(\"error processing shallow info: %d\", reader->status);\n+\n+\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file, NULL);\n+\targs->deepen = 1;\n+}\n+\n enum fetch_state {\n \tFETCH_CHECK_LOCAL = 0,\n \tFETCH_SEND_REQUEST,\n@@ -1205,6 +1266,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t/* v2 supports these by default */\n \t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n \t\t\tuse_sideband = 2;\n+\t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n+\t\t\t\targs->deepen = 1;\n \n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tif (everything_local(args, &ref, sought, nr_sought))\n@@ -1233,6 +1296,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t}\n \t\t\tbreak;\n \t\tcase FETCH_GET_PACK:\n+\t\t\t/* Check for shallow-info section */\n+\t\t\tif (process_section_header(&reader, \"shallow-info\", 1))\n+\t\t\t\treceive_shallow_info(args, &reader);\n+\n \t\t\t/* get the pack */\n \t\t\tprocess_section_header(&reader, \"packfile\", 0);\n \t\t\tif (get_pack(args, fd, pack_lockfile))\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338573","messageId":"20180207011312.189834-28-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 27/35] transport-helper: refactor process_connect_service","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:04Z","receivedAt":"2018-02-07T01:14:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"A future patch will need to take advantage of the logic which runs and\nprocesses the response of the connect command on a remote helper so\nfactor out this logic from 'process_connect_service()' and place it into\na helper function 'run_connect()'.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 67 +++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 38 insertions(+), 29 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex d72155768..c032a2a87 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -545,14 +545,13 @@ static int fetch_with_import(struct transport *transport,\n \treturn 0;\n }\n \n-static int process_connect_service(struct transport *transport,\n-\t\t\t\t const char *name, const char *exec)\n+static int run_connect(struct transport *transport, struct strbuf *cmdbuf)\n {\n \tstruct helper_data *data = transport->data;\n-\tstruct strbuf cmdbuf = STRBUF_INIT;\n-\tstruct child_process *helper;\n-\tint r, duped, ret = 0;\n+\tint ret = 0;\n+\tint duped;\n \tFILE *input;\n+\tstruct child_process *helper;\n \n \thelper = get_helper(transport);\n \n@@ -568,44 +567,54 @@ static int process_connect_service(struct transport *transport,\n \tinput = xfdopen(duped, \"r\");\n \tsetvbuf(input, NULL, _IONBF, 0);\n \n+\tsendline(data, cmdbuf);\n+\tif (recvline_fh(input, cmdbuf))\n+\t\texit(128);\n+\n+\tif (!strcmp(cmdbuf->buf, \"\")) {\n+\t\tdata->no_disconnect_req = 1;\n+\t\tif (debug)\n+\t\t\tfprintf(stderr, \"Debug: Smart transport connection \"\n+\t\t\t\t\"ready.\\n\");\n+\t\tret = 1;\n+\t} else if (!strcmp(cmdbuf->buf, \"fallback\")) {\n+\t\tif (debug)\n+\t\t\tfprintf(stderr, \"Debug: Falling back to dumb \"\n+\t\t\t\t\"transport.\\n\");\n+\t} else {\n+\t\tdie(\"Unknown response to connect: %s\",\n+\t\t\tcmdbuf->buf);\n+\t}\n+\n+\tfclose(input);\n+\treturn ret;\n+}\n+\n+static int process_connect_service(struct transport *transport,\n+\t\t\t\t const char *name, const char *exec)\n+{\n+\tstruct helper_data *data = transport->data;\n+\tstruct strbuf cmdbuf = STRBUF_INIT;\n+\tint ret = 0;\n+\n \t/*\n \t * Handle --upload-pack and friends. This is fire and forget...\n \t * just warn if it fails.\n \t */\n \tif (strcmp(name, exec)) {\n-\t\tr = set_helper_option(transport, \"servpath\", exec);\n+\t\tint r = set_helper_option(transport, \"servpath\", exec);\n \t\tif (r > 0)\n \t\t\twarning(\"Setting remote service path not supported by protocol.\");\n \t\telse if (r < 0)\n \t\t\twarning(\"Invalid remote service path.\");\n \t}\n \n-\tif (data->connect)\n+\tif (data->connect) {\n \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n-\telse\n-\t\tgoto exit;\n-\n-\tsendline(data, &cmdbuf);\n-\tif (recvline_fh(input, &cmdbuf))\n-\t\texit(128);\n-\n-\tif (!strcmp(cmdbuf.buf, \"\")) {\n-\t\tdata->no_disconnect_req = 1;\n-\t\tif (debug)\n-\t\t\tfprintf(stderr, \"Debug: Smart transport connection \"\n-\t\t\t\t\"ready.\\n\");\n-\t\tret = 1;\n-\t} else if (!strcmp(cmdbuf.buf, \"fallback\")) {\n-\t\tif (debug)\n-\t\t\tfprintf(stderr, \"Debug: Falling back to dumb \"\n-\t\t\t\t\"transport.\\n\");\n-\t} else\n-\t\tdie(\"Unknown response to connect: %s\",\n-\t\t\tcmdbuf.buf);\n+\t\tret = run_connect(transport, &cmdbuf);\n+\t}\n \n-exit:\n \tstrbuf_release(&cmdbuf);\n-\tfclose(input);\n \treturn ret;\n }\n \n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338574","messageId":"20180207011312.189834-29-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 28/35] transport-helper: introduce stateless-connect","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:05Z","receivedAt":"2018-02-07T01:14:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the transport-helper capability 'stateless-connect'. This\ncapability indicates that the transport-helper can be requested to run\nthe 'stateless-connect' command which should attempt to make a\nstateless connection with a remote end. Once established, the\nconnection can be used by the git client to communicate with\nthe remote end natively in a stateless-rpc manner as supported by\nprotocol v2. This means that the client must send everything the server\nneeds in a single request as the client must not assume any\nstate-storing on the part of the server or transport.\n\nIf a stateless connection cannot be established then the remote-helper\nwill respond in the same manner as the 'connect' command indicating that\nthe client should fallback to using the dumb remote-helper commands.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 8 ++++++++\n transport.c | 1 +\n transport.h | 6 ++++++\n 3 files changed, 15 insertions(+)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c032a2a87..82eb57c4a 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -26,6 +26,7 @@ struct helper_data {\n \t\toption : 1,\n \t\tpush : 1,\n \t\tconnect : 1,\n+\t\tstateless_connect : 1,\n \t\tsigned_tags : 1,\n \t\tcheck_connectivity : 1,\n \t\tno_disconnect_req : 1,\n@@ -188,6 +189,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\trefspecs[refspec_nr++] = xstrdup(arg);\n \t\t} else if (!strcmp(capname, \"connect\")) {\n \t\t\tdata->connect = 1;\n+\t\t} else if (!strcmp(capname, \"stateless-connect\")) {\n+\t\t\tdata->stateless_connect = 1;\n \t\t} else if (!strcmp(capname, \"signed-tags\")) {\n \t\t\tdata->signed_tags = 1;\n \t\t} else if (skip_prefix(capname, \"export-marks \", &arg)) {\n@@ -612,6 +615,11 @@ static int process_connect_service(struct transport *transport,\n \tif (data->connect) {\n \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n \t\tret = run_connect(transport, &cmdbuf);\n+\t} else if (data->stateless_connect) {\n+\t\tstrbuf_addf(&cmdbuf, \"stateless-connect %s\\n\", name);\n+\t\tret = run_connect(transport, &cmdbuf);\n+\t\tif (ret)\n+\t\t\ttransport->stateless_rpc = 1;\n \t}\n \n \tstrbuf_release(&cmdbuf);\ndiff --git a/transport.c b/transport.c\nindex c275f46ed..9125174f7 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -250,6 +250,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n+\targs.stateless_rpc = transport->stateless_rpc;\n \n \tif (!data->got_remote_heads)\n \t\trefs_tmp = get_refs_via_connect(transport, 0, NULL);\ndiff --git a/transport.h b/transport.h\nindex 4b656f315..9eac809ee 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -55,6 +55,12 @@ struct transport {\n \t */\n \tunsigned cloning : 1;\n \n+\t/*\n+\t * Indicates that the transport is connected via a half-duplex\n+\t * connection and should operate in stateless-rpc mode.\n+\t */\n+\tunsigned stateless_rpc : 1;\n+\n \t/*\n \t * These strings will be passed to the {pre, post}-receive hook,\n \t * on the remote side, if both sides support the push options capability.\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338575","messageId":"20180207011312.189834-30-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 29/35] pkt-line: add packet_buf_write_len function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:06Z","receivedAt":"2018-02-07T01:14:30Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add the 'packet_buf_write_len()' function which allows for writing an\narbitrary length buffer into a 'struct strbuf' and formatting it in\npacket-line format.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 16 ++++++++++++++++\n pkt-line.h | 1 +\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 726e109ca..5a8a17ecc 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -215,6 +215,22 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n \tva_end(args);\n }\n \n+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len)\n+{\n+\tsize_t orig_len, n;\n+\n+\torig_len = buf->len;\n+\tstrbuf_addstr(buf, \"0000\");\n+\tstrbuf_add(buf, data, len);\n+\tn = buf->len - orig_len;\n+\n+\tif (n > LARGE_PACKET_MAX)\n+\t\tdie(\"protocol error: impossibly long line\");\n+\n+\tset_packet_header(&buf->buf[orig_len], n);\n+\tpacket_trace(buf->buf + orig_len + 4, n - 4, 1);\n+}\n+\n int write_packetized_from_fd(int fd_in, int fd_out)\n {\n \tstatic char buf[LARGE_PACKET_DATA_MAX];\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 16fe8bdbf..63724d4bf 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -26,6 +26,7 @@ void packet_buf_flush(struct strbuf *buf);\n void packet_buf_delim(struct strbuf *buf);\n void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len);\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int write_packetized_from_fd(int fd_in, int fd_out);\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338576","messageId":"20180207011312.189834-31-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 30/35] remote-curl: create copy of the service name","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:07Z","receivedAt":"2018-02-07T01:14:34Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Make a copy of the service name being requested instead of relying on\nthe buffer pointed to by the passed in 'const char *' to remain\nunchanged.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex dae8a4a48..4086aa733 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -165,7 +165,7 @@ static int set_option(const char *name, const char *value)\n }\n \n struct discovery {\n-\tconst char *service;\n+\tchar *service;\n \tchar *buf_alloc;\n \tchar *buf;\n \tsize_t len;\n@@ -257,6 +257,7 @@ static void free_discovery(struct discovery *d)\n \t\tfree(d->shallow.oid);\n \t\tfree(d->buf_alloc);\n \t\tfree_refs(d->refs);\n+\t\tfree(d->service);\n \t\tfree(d);\n \t}\n }\n@@ -343,7 +344,7 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \t\twarning(_(\"redirecting to %s\"), url.buf);\n \n \tlast= xcalloc(1, sizeof(*last_discovery));\n-\tlast->service = service;\n+\tlast->service = xstrdup(service);\n \tlast->buf_alloc = strbuf_detach(&buffer, &last->len);\n \tlast->buf = last->buf_alloc;\n \n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338577","messageId":"20180207011312.189834-33-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 32/35] http: allow providing extra headers for http requests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:09Z","receivedAt":"2018-02-07T01:14:37Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a way for callers to request that extra headers be included when\nmaking http requests.\n\nSigned-off-by: Brandon Williams \n---\n http.c | 8 ++++++++\n http.h | 2 ++\n 2 files changed, 10 insertions(+)\n\ndiff --git a/http.c b/http.c\nindex 597771271..e1757d62b 100644\n--- a/http.c\n+++ b/http.c\n@@ -1723,6 +1723,14 @@ static int http_request(const char *url,\n \n \theaders = curl_slist_append(headers, buf.buf);\n \n+\t/* Add additional headers here */\n+\tif (options && options->extra_headers) {\n+\t\tconst struct string_list_item *item;\n+\t\tfor_each_string_list_item(item, options->extra_headers) {\n+\t\t\theaders = curl_slist_append(headers, item->string);\n+\t\t}\n+\t}\n+\n \tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, \"gzip\");\ndiff --git a/http.h b/http.h\nindex f7bd3b26b..a113915c7 100644\n--- a/http.h\n+++ b/http.h\n@@ -172,6 +172,8 @@ struct http_get_options {\n \t * for details.\n \t */\n \tstruct strbuf *base_url;\n+\n+\tstruct string_list *extra_headers;\n };\n \n /* Return values for http_get_*() */\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338578","messageId":"20180207011312.189834-34-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 33/35] http: don't always add Git-Protocol header","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:10Z","receivedAt":"2018-02-07T01:14:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Instead of always sending the Git-Protocol header with the configured\nversion with every http request, explicitly send it when discovering\nrefs and then only send it on subsequent http requests if the server\nunderstood the version requested.\n\nSigned-off-by: Brandon Williams \n---\n http.c | 17 -----------------\n remote-curl.c | 33 +++++++++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 17 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex e1757d62b..8f1129ac7 100644\n--- a/http.c\n+++ b/http.c\n@@ -904,21 +904,6 @@ static void set_from_env(const char **var, const char *envname)\n \t\t*var = val;\n }\n \n-static void protocol_http_header(void)\n-{\n-\tif (get_protocol_version_config() > 0) {\n-\t\tstruct strbuf protocol_header = STRBUF_INIT;\n-\n-\t\tstrbuf_addf(&protocol_header, GIT_PROTOCOL_HEADER \": version=%d\",\n-\t\t\t get_protocol_version_config());\n-\n-\n-\t\textra_http_headers = curl_slist_append(extra_http_headers,\n-\t\t\t\t\t\t protocol_header.buf);\n-\t\tstrbuf_release(&protocol_header);\n-\t}\n-}\n-\n void http_init(struct remote *remote, const char *url, int proactive_auth)\n {\n \tchar *low_speed_limit;\n@@ -949,8 +934,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tif (remote)\n \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n \n-\tprotocol_http_header();\n-\n \tpragma_header = curl_slist_append(http_copy_default_headers(),\n \t\t\"Pragma: no-cache\");\n \tno_pragma_header = curl_slist_append(http_copy_default_headers(),\ndiff --git a/remote-curl.c b/remote-curl.c\nindex c54035843..b4e9db85b 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -291,6 +291,19 @@ static int show_http_message(struct strbuf *type, struct strbuf *charset,\n \treturn 0;\n }\n \n+static int get_protocol_http_header(enum protocol_version version,\n+\t\t\t\t struct strbuf *header)\n+{\n+\tif (version > 0) {\n+\t\tstrbuf_addf(header, GIT_PROTOCOL_HEADER \": version=%d\",\n+\t\t\t version);\n+\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static struct discovery *discover_refs(const char *service, int for_push)\n {\n \tstruct strbuf exp = STRBUF_INIT;\n@@ -299,6 +312,8 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \tstruct strbuf buffer = STRBUF_INIT;\n \tstruct strbuf refs_url = STRBUF_INIT;\n \tstruct strbuf effective_url = STRBUF_INIT;\n+\tstruct strbuf protocol_header = STRBUF_INIT;\n+\tstruct string_list extra_headers = STRING_LIST_INIT_DUP;\n \tstruct discovery *last = last_discovery;\n \tint http_ret, maybe_smart = 0;\n \tstruct http_get_options http_options;\n@@ -318,11 +333,16 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \t\tstrbuf_addf(&refs_url, \"service=%s\", service);\n \t}\n \n+\t/* Add the extra Git-Protocol header */\n+\tif (get_protocol_http_header(get_protocol_version_config(), &protocol_header))\n+\t\tstring_list_append(&extra_headers, protocol_header.buf);\n+\n \tmemset(&http_options, 0, sizeof(http_options));\n \thttp_options.content_type = &type;\n \thttp_options.charset = &charset;\n \thttp_options.effective_url = &effective_url;\n \thttp_options.base_url = &url;\n+\thttp_options.extra_headers = &extra_headers;\n \thttp_options.initial_request = 1;\n \thttp_options.no_cache = 1;\n \thttp_options.keep_error = 1;\n@@ -389,6 +409,8 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \tstrbuf_release(&charset);\n \tstrbuf_release(&effective_url);\n \tstrbuf_release(&buffer);\n+\tstrbuf_release(&protocol_header);\n+\tstring_list_clear(&extra_headers, 0);\n \tlast_discovery = last;\n \treturn last;\n }\n@@ -425,6 +447,7 @@ struct rpc_state {\n \tchar *service_url;\n \tchar *hdr_content_type;\n \tchar *hdr_accept;\n+\tchar *protocol_header;\n \tchar *buf;\n \tsize_t alloc;\n \tsize_t len;\n@@ -611,6 +634,10 @@ static int post_rpc(struct rpc_state *rpc)\n \theaders = curl_slist_append(headers, needs_100_continue ?\n \t\t\"Expect: 100-continue\" : \"Expect:\");\n \n+\t/* Add the extra Git-Protocol header */\n+\tif (rpc->protocol_header)\n+\t\theaders = curl_slist_append(headers, rpc->protocol_header);\n+\n retry:\n \tslot = get_active_slot();\n \n@@ -751,6 +778,11 @@ static int rpc_service(struct rpc_state *rpc, struct discovery *heads)\n \tstrbuf_addf(&buf, \"Accept: application/x-%s-result\", svc);\n \trpc->hdr_accept = strbuf_detach(&buf, NULL);\n \n+\tif (get_protocol_http_header(heads->version, &buf))\n+\t\trpc->protocol_header = strbuf_detach(&buf, NULL);\n+\telse\n+\t\trpc->protocol_header = NULL;\n+\n \twhile (!err) {\n \t\tint n = packet_read(rpc->out, NULL, NULL, rpc->buf, rpc->alloc, 0);\n \t\tif (!n)\n@@ -778,6 +810,7 @@ static int rpc_service(struct rpc_state *rpc, struct discovery *heads)\n \tfree(rpc->service_url);\n \tfree(rpc->hdr_content_type);\n \tfree(rpc->hdr_accept);\n+\tfree(rpc->protocol_header);\n \tfree(rpc->buf);\n \tstrbuf_release(&buf);\n \treturn err;\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338579","messageId":"20180207011312.189834-35-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 34/35] remote-curl: implement stateless-connect command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:11Z","receivedAt":"2018-02-07T01:14:42Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach remote-curl the 'stateless-connect' command which is used to\nestablish a stateless connection with servers which support protocol\nversion 2. This allows remote-curl to act as a proxy, allowing the git\nclient to communicate natively with a remote end, simply using\nremote-curl as a pass through to convert requests to http.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 187 ++++++++++++++++++++++++++++++++++++++++++++++++-\n t/t5702-protocol-v2.sh | 41 +++++++++++\n 2 files changed, 227 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex b4e9db85b..af431b658 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -188,7 +188,10 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \theads->version = discover_version(&reader);\n \tswitch (heads->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\t/*\n+\t\t * Do nothing. Client should run 'stateless-connect' and\n+\t\t * request the refs themselves.\n+\t\t */\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n@@ -1082,6 +1085,184 @@ static void parse_push(struct strbuf *buf)\n \tfree(specs);\n }\n \n+struct proxy_state {\n+\tchar *service_name;\n+\tchar *service_url;\n+\tstruct curl_slist *headers;\n+\tstruct strbuf request_buffer;\n+\tint in;\n+\tint out;\n+\tstruct packet_reader reader;\n+\tsize_t pos;\n+\tint seen_flush;\n+};\n+\n+static void proxy_state_init(struct proxy_state *p, const char *service_name,\n+\t\t\t enum protocol_version version)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tmemset(p, 0, sizeof(*p));\n+\tp->service_name = xstrdup(service_name);\n+\n+\tp->in = 0;\n+\tp->out = 1;\n+\tstrbuf_init(&p->request_buffer, 0);\n+\n+\tstrbuf_addf(&buf, \"%s%s\", url.buf, p->service_name);\n+\tp->service_url = strbuf_detach(&buf, NULL);\n+\n+\tp->headers = http_copy_default_headers();\n+\n+\tstrbuf_addf(&buf, \"Content-Type: application/x-%s-request\", p->service_name);\n+\tp->headers = curl_slist_append(p->headers, buf.buf);\n+\tstrbuf_reset(&buf);\n+\n+\tstrbuf_addf(&buf, \"Accept: application/x-%s-result\", p->service_name);\n+\tp->headers = curl_slist_append(p->headers, buf.buf);\n+\tstrbuf_reset(&buf);\n+\n+\tp->headers = curl_slist_append(p->headers, \"Transfer-Encoding: chunked\");\n+\n+\t/* Add the Git-Protocol header */\n+\tif (get_protocol_http_header(version, &buf))\n+\t\tp->headers = curl_slist_append(p->headers, buf.buf);\n+\n+\tpacket_reader_init(&p->reader, p->in, NULL, 0,\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n+static void proxy_state_clear(struct proxy_state *p)\n+{\n+\tfree(p->service_name);\n+\tfree(p->service_url);\n+\tcurl_slist_free_all(p->headers);\n+\tstrbuf_release(&p->request_buffer);\n+}\n+\n+static size_t proxy_in(char *buffer, size_t eltsize,\n+\t\t size_t nmemb, void *userdata)\n+{\n+\tsize_t max = eltsize * nmemb;\n+\tstruct proxy_state *p = userdata;\n+\tsize_t avail = p->request_buffer.len - p->pos;\n+\n+\tif (!avail) {\n+\t\tif (p->seen_flush) {\n+\t\t\tp->seen_flush = 0;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tstrbuf_reset(&p->request_buffer);\n+\t\tswitch (packet_reader_read(&p->reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tdie(\"unexpected EOF when reading from parent process\");\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tpacket_buf_write_len(&p->request_buffer, p->reader.line,\n+\t\t\t\t\t p->reader.pktlen);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tpacket_buf_delim(&p->request_buffer);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tpacket_buf_flush(&p->request_buffer);\n+\t\t\tp->seen_flush = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tp->pos = 0;\n+\t\tavail = p->request_buffer.len;\n+\t}\n+\n+\tif (max < avail)\n+\t\tavail = max;\n+\tmemcpy(buffer, p->request_buffer.buf + p->pos, avail);\n+\tp->pos += avail;\n+\treturn avail;\n+}\n+\n+static size_t proxy_out(char *buffer, size_t eltsize,\n+\t\t\tsize_t nmemb, void *userdata)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct proxy_state *p = userdata;\n+\n+\twrite_or_die(p->out, buffer, size);\n+\treturn size;\n+}\n+\n+static int proxy_post(struct proxy_state *p)\n+{\n+\tstruct active_request_slot *slot;\n+\tint err;\n+\n+\tslot = get_active_slot();\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, p->headers);\n+\n+\t/* Setup function to read request from client */\n+\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_READDATA, p);\n+\n+\t/* Setup function to write server response to client */\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);\n+\n+\terr = run_slot(slot, NULL);\n+\n+\tif (err != HTTP_OK)\n+\t\terr = -1;\n+\n+\treturn err;\n+}\n+\n+static int stateless_connect(const char *service_name)\n+{\n+\tstruct discovery *discover;\n+\tstruct proxy_state p;\n+\n+\t/*\n+\t * Run the info/refs request and see if the server supports protocol\n+\t * v2. If and only if the server supports v2 can we successfully\n+\t * establish a stateless connection, otherwise we need to tell the\n+\t * client to fallback to using other transport helper functions to\n+\t * complete their request.\n+\t */\n+\tdiscover = discover_refs(service_name, 0);\n+\tif (discover->version != protocol_v2) {\n+\t\tprintf(\"fallback\\n\");\n+\t\tfflush(stdout);\n+\t\treturn -1;\n+\t} else {\n+\t\t/* Stateless Connection established */\n+\t\tprintf(\"\\n\");\n+\t\tfflush(stdout);\n+\t}\n+\n+\tproxy_state_init(&p, service_name, discover->version);\n+\n+\t/*\n+\t * Dump the capability listing that we got from the server earlier\n+\t * during the info/refs request.\n+\t */\n+\twrite_or_die(p.out, discover->buf, discover->len);\n+\n+\t/* Peek the next packet line. Until we see EOF keep sending POSTs */\n+\twhile (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {\n+\t\tif (proxy_post(&p)) {\n+\t\t\t/* We would have an err here */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tproxy_state_clear(&p);\n+\treturn 0;\n+}\n+\n int cmd_main(int argc, const char **argv)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -1150,12 +1331,16 @@ int cmd_main(int argc, const char **argv)\n \t\t\tfflush(stdout);\n \n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n+\t\t\tprintf(\"stateless-connect\\n\");\n \t\t\tprintf(\"fetch\\n\");\n \t\t\tprintf(\"option\\n\");\n \t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"check-connectivity\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n+\t\t} else if (skip_prefix(buf.buf, \"stateless-connect \", &arg)) {\n+\t\t\tif (!stateless_connect(arg))\n+\t\t\t\tbreak;\n \t\t} else {\n \t\t\terror(\"remote-curl: unknown command '%s' from git\", buf.buf);\n \t\t\treturn 1;\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 60e43bcf5..c2c39fe0c 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -172,4 +172,45 @@ test_expect_success 'ref advertisment is filtered during fetch using protocol v2\n \t! grep \"refs/tags/\" log\n '\n \n+# Test protocol v2 with 'http://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one\n+'\n+\n+test_expect_success 'clone with http:// using protocol v2' '\n+\tGIT_TRACE_PACKET=1 GIT_TRACE_CURL=1 git -c protocol.version=2 \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"Git-Protocol: version=2\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log\n+'\n+\n+test_expect_success 'fetch with http:// using protocol v2' '\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=2 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338580","messageId":"20180207011312.189834-25-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 24/35] connect: refactor git_connect to only get the protocol version once","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:01Z","receivedAt":"2018-02-07T01:14:45Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Instead of having each builtin transport asking for which protocol\nversion the user has configured in 'protocol.version' by calling\n`get_protocol_version_config()` multiple times, factor this logic out\nso there is just a single call at the beginning of `git_connect()`.\n\nThis will be helpful in the next patch where we can have centralized\nlogic which determines if we need to request a different protocol\nversion than what the user has configured.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 27 +++++++++++++++------------\n 1 file changed, 15 insertions(+), 12 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 9577528f3..dbf4def65 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -1029,6 +1029,7 @@ static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n */\n static struct child_process *git_connect_git(int fd[2], char *hostandport,\n \t\t\t\t\t const char *path, const char *prog,\n+\t\t\t\t\t enum protocol_version version,\n \t\t\t\t\t int flags)\n {\n \tstruct child_process *conn;\n@@ -1067,10 +1068,10 @@ static struct child_process *git_connect_git(int fd[2], char *hostandport,\n \t\t target_host, 0);\n \n \t/* If using a new version put that stuff here after a second null byte */\n-\tif (get_protocol_version_config() > 0) {\n+\tif (version > 0) {\n \t\tstrbuf_addch(&request, '\\0');\n \t\tstrbuf_addf(&request, \"version=%d%c\",\n-\t\t\t get_protocol_version_config(), '\\0');\n+\t\t\t version, '\\0');\n \t}\n \n \tpacket_write(fd[1], request.buf, request.len);\n@@ -1086,14 +1087,14 @@ static struct child_process *git_connect_git(int fd[2], char *hostandport,\n */\n static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n \t\t\t enum ssh_variant variant, const char *port,\n-\t\t\t int flags)\n+\t\t\t enum protocol_version version, int flags)\n {\n \tif (variant == VARIANT_SSH &&\n-\t get_protocol_version_config() > 0) {\n+\t version > 0) {\n \t\targv_array_push(args, \"-o\");\n \t\targv_array_push(args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n \t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n-\t\t\t\t get_protocol_version_config());\n+\t\t\t\t version);\n \t}\n \n \tif (flags & CONNECT_IPV4) {\n@@ -1146,7 +1147,8 @@ static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n \n /* Prepare a child_process for use by Git's SSH-tunneled transport. */\n static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n-\t\t\t const char *port, int flags)\n+\t\t\t const char *port, enum protocol_version version,\n+\t\t\t int flags)\n {\n \tconst char *ssh;\n \tenum ssh_variant variant;\n@@ -1180,14 +1182,14 @@ static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n \t\targv_array_push(&detect.args, ssh);\n \t\targv_array_push(&detect.args, \"-G\");\n \t\tpush_ssh_options(&detect.args, &detect.env_array,\n-\t\t\t\t VARIANT_SSH, port, flags);\n+\t\t\t\t VARIANT_SSH, port, version, flags);\n \t\targv_array_push(&detect.args, ssh_host);\n \n \t\tvariant = run_command(&detect) ? VARIANT_SIMPLE : VARIANT_SSH;\n \t}\n \n \targv_array_push(&conn->args, ssh);\n-\tpush_ssh_options(&conn->args, &conn->env_array, variant, port, flags);\n+\tpush_ssh_options(&conn->args, &conn->env_array, variant, port, version, flags);\n \targv_array_push(&conn->args, ssh_host);\n }\n \n@@ -1208,6 +1210,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \tchar *hostandport, *path;\n \tstruct child_process *conn;\n \tenum protocol protocol;\n+\tenum protocol_version version = get_protocol_version_config();\n \n \t/* Without this we cannot rely on waitpid() to tell\n \t * what happened to our children.\n@@ -1222,7 +1225,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tprintf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n \t\tconn = NULL;\n \t} else if (protocol == PROTO_GIT) {\n-\t\tconn = git_connect_git(fd, hostandport, path, prog, flags);\n+\t\tconn = git_connect_git(fd, hostandport, path, prog, version, flags);\n \t} else {\n \t\tstruct strbuf cmd = STRBUF_INIT;\n \t\tconst char *const *var;\n@@ -1265,12 +1268,12 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tstrbuf_release(&cmd);\n \t\t\t\treturn NULL;\n \t\t\t}\n-\t\t\tfill_ssh_args(conn, ssh_host, port, flags);\n+\t\t\tfill_ssh_args(conn, ssh_host, port, version, flags);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\n-\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\tif (version > 0) {\n \t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n-\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t\t\t\t version);\n \t\t\t}\n \t\t}\n \t\targv_array_push(&conn->args, cmd.buf);\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338581","messageId":"20180207011312.189834-36-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 35/35] remote-curl: don't request v2 when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:12Z","receivedAt":"2018-02-07T01:14:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to be able to ship protocol v2 with only supporting fetch, we\nneed clients to not issue a request to use protocol v2 when pushing\n(since the client currently doesn't know how to push using protocol v2).\nThis allows a client to have protocol v2 configured in\n`protocol.version` and take advantage of using v2 for fetch and falling\nback to using v0 when pushing while v2 for push is being designed.\n\nWe could run into issues if we didn't fall back to protocol v2 when\npushing right now. This is because currently a server will ignore a request to\nuse v2 when contacting the 'receive-pack' endpoint and fall back to\nusing v0, but when push v2 is rolled out to servers, the 'receive-pack'\nendpoint will start responding using v2. So we don't want to get into a\nstate where a client is requesting to push with v2 before they actually\nknow how to push using v2.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 11 ++++++++++-\n t/t5702-protocol-v2.sh | 23 +++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex af431b658..c39b6ece6 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -320,6 +320,7 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \tstruct discovery *last = last_discovery;\n \tint http_ret, maybe_smart = 0;\n \tstruct http_get_options http_options;\n+\tenum protocol_version version = get_protocol_version_config();\n \n \tif (last && !strcmp(service, last->service))\n \t\treturn last;\n@@ -336,8 +337,16 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \t\tstrbuf_addf(&refs_url, \"service=%s\", service);\n \t}\n \n+\t/*\n+\t * NEEDSWORK: If we are trying to use protocol v2 and we are planning\n+\t * to perform a push, then fallback to v0 since the client doesn't know\n+\t * how to push yet using v2.\n+\t */\n+\tif (version == protocol_v2 && !strcmp(\"git-receive-pack\", service))\n+\t\tversion = protocol_v0;\n+\n \t/* Add the extra Git-Protocol header */\n-\tif (get_protocol_http_header(get_protocol_version_config(), &protocol_header))\n+\tif (get_protocol_http_header(version, &protocol_header))\n \t\tstring_list_append(&extra_headers, protocol_header.buf);\n \n \tmemset(&http_options, 0, sizeof(http_options));\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex c2c39fe0c..14d589a7f 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -211,6 +211,29 @@ test_expect_success 'fetch with http:// using protocol v2' '\n \tgrep \"git< version 2\" log\n '\n \n+test_expect_success 'push with http:// and a config of v2 does not request v2' '\n+\t# Till v2 for push is designed, make sure that if a client has\n+\t# protocol.version configured to use v2, that the client instead falls\n+\t# back and uses v0.\n+\n+\ttest_commit -C http_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch && 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client didnt request to use protocol v2\n+\t! grep \"Git-Protocol: version=2\" log &&\n+\t# Server didnt respond using protocol v2\n+\t! grep \"git< version 2\" log\n+'\n+\n+\n stop_httpd\n \n test_done\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338582","messageId":"20180207011312.189834-32-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 31/35] remote-curl: store the protocol version the server responded with","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:08Z","receivedAt":"2018-02-07T01:14:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Store the protocol version the server responded with when performing\ndiscovery. This will be used in a future patch to either change the\n'Git-Protocol' header sent in subsequent requests or to determine if a\nclient needs to fallback to using a different protocol version.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 4086aa733..c54035843 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -171,6 +171,7 @@ struct discovery {\n \tsize_t len;\n \tstruct ref *refs;\n \tstruct oid_array shallow;\n+\tenum protocol_version version;\n \tunsigned proto_git : 1;\n };\n static struct discovery *last_discovery;\n@@ -184,7 +185,8 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n-\tswitch (discover_version(&reader)) {\n+\theads->version = discover_version(&reader);\n+\tswitch (heads->version) {\n \tcase protocol_v2:\n \t\tdie(\"support for protocol v2 not implemented yet\");\n \t\tbreak;\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338583","messageId":"20180207011312.189834-26-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 25/35] connect: don't request v2 when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:02Z","receivedAt":"2018-02-07T01:14:53Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to be able to ship protocol v2 with only supporting fetch, we\nneed clients to not issue a request to use protocol v2 when pushing\n(since the client currently doesn't know how to push using protocol v2).\nThis allows a client to have protocol v2 configured in\n`protocol.version` and take advantage of using v2 for fetch and falling\nback to using v0 when pushing while v2 for push is being designed.\n\nWe could run into issues if we didn't fall back to protocol v2 when\npushing right now. This is because currently a server will ignore a request to\nuse v2 when contacting the 'receive-pack' endpoint and fall back to\nusing v0, but when push v2 is rolled out to servers, the 'receive-pack'\nendpoint will start responding using v2. So we don't want to get into a\nstate where a client is requesting to push with v2 before they actually\nknow how to push using v2.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 8 ++++++++\n t/t5702-protocol-v2.sh | 22 ++++++++++++++++++++++\n 2 files changed, 30 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex dbf4def65..37a6a8935 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -1212,6 +1212,14 @@ struct child_process *git_connect(int fd[2], const char *url,\n \tenum protocol protocol;\n \tenum protocol_version version = get_protocol_version_config();\n \n+\t/*\n+\t * NEEDSWORK: If we are trying to use protocol v2 and we are planning\n+\t * to perform a push, then fallback to v0 since the client doesn't know\n+\t * how to push yet using v2.\n+\t */\n+\tif (version == protocol_v2 && !strcmp(\"git-receive-pack\", prog))\n+\t\tversion = protocol_v0;\n+\n \t/* Without this we cannot rely on waitpid() to tell\n \t * what happened to our children.\n \t */\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 16304d1c8..60e43bcf5 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -82,6 +82,28 @@ test_expect_success 'pull with git:// using protocol v2' '\n \tgrep \"fetch< version 2\" log\n '\n \n+test_expect_success 'push with git:// and a config of v2 does not request v2' '\n+\t# Till v2 for push is designed, make sure that if a client has\n+\t# protocol.version configured to use v2, that the client instead falls\n+\t# back and uses v0.\n+\n+\ttest_commit -C daemon_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=2 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\t! grep \"push> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\t! grep \"push< version 2\" log\n+'\n+\n stop_git_daemon\n \n # Test protocol v2 with 'file://' transport\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338584","messageId":"20180207011312.189834-27-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 26/35] transport-helper: remove name parameter","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:13:03Z","receivedAt":"2018-02-07T01:14:55Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Commit 266f1fdfa (transport-helper: be quiet on read errors from\nhelpers, 2013-06-21) removed a call to 'die()' which printed the name of\nthe remote helper passed in to the 'recvline_fh()' function using the\n'name' parameter. Once the call to 'die()' was removed the parameter\nwas no longer necessary but wasn't removed. Clean up 'recvline_fh()'\nparameter list by removing the 'name' parameter.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 4c334b5ee..d72155768 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -49,7 +49,7 @@ static void sendline(struct helper_data *helper, struct strbuf *buffer)\n \t\tdie_errno(\"Full write to remote helper failed\");\n }\n \n-static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)\n+static int recvline_fh(FILE *helper, struct strbuf *buffer)\n {\n \tstrbuf_reset(buffer);\n \tif (debug)\n@@ -67,7 +67,7 @@ static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)\n \n static int recvline(struct helper_data *helper, struct strbuf *buffer)\n {\n-\treturn recvline_fh(helper->out, buffer, helper->name);\n+\treturn recvline_fh(helper->out, buffer);\n }\n \n static void write_constant(int fd, const char *str)\n@@ -586,7 +586,7 @@ static int process_connect_service(struct transport *transport,\n \t\tgoto exit;\n \n \tsendline(data, &cmdbuf);\n-\tif (recvline_fh(input, &cmdbuf, name))\n+\tif (recvline_fh(input, &cmdbuf))\n \t\texit(128);\n \n \tif (!strcmp(cmdbuf.buf, \"\")) {\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338585","messageId":"20180207011312.189834-16-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 15/35] transport: convert get_refs_list to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:52Z","receivedAt":"2018-02-07T01:15:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert the 'struct transport' virtual function 'get_refs_list()' to\noptionally take an argv_array of ref patterns. When communicating with\na server using protocol v2 these ref patterns can be sent when\nrequesting a listing of their refs allowing the server to filter the\nrefs it sends based on the sent patterns.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 5 +++--\n transport-internal.h | 4 +++-\n transport.c | 16 +++++++++-------\n 3 files changed, 15 insertions(+), 10 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 508015023..4c334b5ee 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1026,7 +1026,8 @@ static int has_attribute(const char *attrs, const char *attr) {\n \t}\n }\n \n-static struct ref *get_refs_list(struct transport *transport, int for_push)\n+static struct ref *get_refs_list(struct transport *transport, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns)\n {\n \tstruct helper_data *data = transport->data;\n \tstruct child_process *helper;\n@@ -1039,7 +1040,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)\n \n \tif (process_connect(transport, for_push)) {\n \t\tdo_take_over(transport);\n-\t\treturn transport->vtable->get_refs_list(transport, for_push);\n+\t\treturn transport->vtable->get_refs_list(transport, for_push, ref_patterns);\n \t}\n \n \tif (data->push && for_push)\ndiff --git a/transport-internal.h b/transport-internal.h\nindex 3c1a29d72..a67657ce3 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -3,6 +3,7 @@\n \n struct ref;\n struct transport;\n+struct argv_array;\n \n struct transport_vtable {\n \t/**\n@@ -21,7 +22,8 @@ struct transport_vtable {\n \t * the ref without a huge amount of effort, it should store it\n \t * in the ref's old_sha1 field; otherwise it should be all 0.\n \t **/\n-\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push);\n+\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns);\n \n \t/**\n \t * Fetch the objects for the given refs. Note that this gets\ndiff --git a/transport.c b/transport.c\nindex ffc6b2614..c54a44630 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -72,7 +72,7 @@ struct bundle_transport_data {\n \tstruct bundle_header header;\n };\n \n-static struct ref *get_refs_from_bundle(struct transport *transport, int for_push)\n+static struct ref *get_refs_from_bundle(struct transport *transport, int for_push, const struct argv_array *ref_patterns)\n {\n \tstruct bundle_transport_data *data = transport->data;\n \tstruct ref *result = NULL;\n@@ -189,7 +189,8 @@ static int connect_setup(struct transport *transport, int for_push)\n \treturn 0;\n }\n \n-static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n+static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n+\t\t\t\t\tconst struct argv_array *ref_patterns)\n {\n \tstruct git_transport_data *data = transport->data;\n \tstruct ref *refs = NULL;\n@@ -204,7 +205,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);\n+\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push,\n+\t\t\t\tref_patterns);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n@@ -250,7 +252,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.update_shallow = data->options.update_shallow;\n \n \tif (!data->got_remote_heads)\n-\t\trefs_tmp = get_refs_via_connect(transport, 0);\n+\t\trefs_tmp = get_refs_via_connect(transport, 0, NULL);\n \n \tswitch (data->version) {\n \tcase protocol_v2:\n@@ -568,7 +570,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tint ret = 0;\n \n \tif (!data->got_remote_heads)\n-\t\tget_refs_via_connect(transport, 1);\n+\t\tget_refs_via_connect(transport, 1, NULL);\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n@@ -1028,7 +1030,7 @@ int transport_push(struct transport *transport,\n \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n \t\t\treturn -1;\n \n-\t\tremote_refs = transport->vtable->get_refs_list(transport, 1);\n+\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n \n \t\tif (flags & TRANSPORT_PUSH_ALL)\n \t\t\tmatch_flags |= MATCH_REFS_ALL;\n@@ -1137,7 +1139,7 @@ int transport_push(struct transport *transport,\n const struct ref *transport_get_remote_refs(struct transport *transport)\n {\n \tif (!transport->got_remote_refs) {\n-\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0);\n+\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n \t\ttransport->got_remote_refs = 1;\n \t}\n \n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338586","messageId":"20180207011312.189834-21-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 20/35] upload-pack: introduce fetch server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:57Z","receivedAt":"2018-02-07T01:15:05Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the 'fetch' server command.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/technical/protocol-v2.txt | 127 +++++++++++++++\n serve.c | 2 +\n t/t5701-git-serve.sh | 1 +\n upload-pack.c | 281 ++++++++++++++++++++++++++++++++\n upload-pack.h | 5 +\n 5 files changed, 416 insertions(+)\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex ef81df868..4d5096dae 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -144,3 +144,130 @@ The output of ls-refs is as follows:\n ref-attribute = (symref | peeled)\n symref = \"symref-target:\" symref-target\n peeled = \"peeled:\" obj-id\n+\n+ fetch\n+-------\n+\n+`fetch` is the command used to fetch a packfile in v2. It can be looked\n+at as a modified version of the v1 fetch where the ref-advertisement is\n+stripped out (since the `ls-refs` command fills that role) and the\n+message format is tweaked to eliminate redundancies and permit easy\n+addition of future extensions.\n+\n+Additional features not supported in the base command will be advertised\n+as the value of the command in the capability advertisement in the form\n+of a space separated list of features, e.g. \"=\n+\".\n+\n+A `fetch` request can take the following parameters wrapped in\n+packet-lines:\n+\n+ want \n+\tIndicates to the server an object which the client wants to\n+\tretrieve.\n+\n+ have \n+\tIndicates to the server an object which the client has locally.\n+\tThis allows the server to make a packfile which only contains\n+\tthe objects that the client needs. Multiple 'have' lines can be\n+\tsupplied.\n+\n+ done\n+\tIndicates to the server that negotiation should terminate (or\n+\tnot even begin if performing a clone) and that the server should\n+\tuse the information supplied in the request to construct the\n+\tpackfile.\n+\n+ thin-pack\n+\tRequest that a thin pack be sent, which is a pack with deltas\n+\twhich reference base objects not contained within the pack (but\n+\tare known to exist at the receiving end). This can reduce the\n+\tnetwork traffic significantly, but it requires the receiving end\n+\tto know how to \"thicken\" these packs by adding the missing bases\n+\tto the pack.\n+\n+ no-progress\n+\tRequest that progress information that would normally be sent on\n+\tside-band channel 2, during the packfile transfer, should not be\n+\tsent. However, the side-band channel 3 is still used for error\n+\tresponses.\n+\n+ include-tag\n+\tRequest that annotated tags should be sent if the objects they\n+\tpoint to are being sent.\n+\n+ ofs-delta\n+\tIndicate that the client understands PACKv2 with delta referring\n+\tto its base by position in pack rather than by an oid. That is,\n+\tthey can read OBJ_OFS_DELTA (ake type 6) in a packfile.\n+\n+The response of `fetch` is broken into a number of sections separated by\n+delimiter packets (0001), with each section beginning with its section\n+header.\n+\n+ output = *section\n+ section = (acknowledgments | packfile)\n+\t (flush-pkt | delim-pkt)\n+\n+ acknowledgments = PKT-LINE(\"acknowledgments\" LF)\n+\t\t *(ready | nak | ack)\n+ ready = PKT-LINE(\"ready\" LF)\n+ nak = PKT-LINE(\"NAK\" LF)\n+ ack = PKT-LINE(\"ACK\" SP obj-id LF)\n+\n+ packfile = PKT-LINE(\"packfile\" LF)\n+\t [PACKFILE]\n+\n+----\n+ acknowledgments section\n+\t* Always begins with the section header \"acknowledgments\"\n+\n+\t* The server will respond with \"NAK\" if none of the object ids sent\n+\t as have lines were common.\n+\n+\t* The server will respond with \"ACK obj-id\" for all of the\n+\t object ids sent as have lines which are common.\n+\n+\t* A response cannot have both \"ACK\" lines as well as a \"NAK\"\n+\t line.\n+\n+\t* The server will respond with a \"ready\" line indicating that\n+\t the server has found an acceptable common base and is ready to\n+\t make and send a packfile (which will be found in the packfile\n+\t section of the same response)\n+\n+\t* If the client determines that it is finished with negotiations\n+\t by sending a \"done\" line, the acknowledgments sections can be\n+\t omitted from the server's response as an optimization.\n+\n+\t* If the server has found a suitable cut point and has decided\n+\t to send a \"ready\" line, then the server can decide to (as an\n+\t optimization) omit any \"ACK\" lines it would have sent during\n+\t its response. This is because the server will have already\n+\t determined the objects it plans to send to the client and no\n+\t further negotiation is needed.\n+\n+----\n+ packfile section\n+\t* Always begins with the section header \"packfile\"\n+\n+\t* The transmission of the packfile begins immediately after the\n+\t section header\n+\n+\t* The data transfer of the packfile is always multiplexed, using\n+\t the same semantics of the 'side-band-64k' capability from\n+\t protocol version 1. This means that each packet, during the\n+\t packfile data stream, is made up of a leading 4-byte pkt-line\n+\t length (typical of the pkt-line format), followed by a 1-byte\n+\t stream code, followed by the actual data.\n+\n+\t The stream code can be one of:\n+\t\t1 - pack data\n+\t\t2 - progress messages\n+\t\t3 - fatal error message just before stream aborts\n+\n+\t* This section is only included if the client has sent 'want'\n+\t lines in its request and either requested that no more\n+\t negotiation be done by sending 'done' or if the server has\n+\t decided it has found a sufficient cut point to produce a\n+\t packfile.\ndiff --git a/serve.c b/serve.c\nindex c7925c5c7..05cc434cf 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -6,6 +6,7 @@\n #include \"argv-array.h\"\n #include \"ls-refs.h\"\n #include \"serve.h\"\n+#include \"upload-pack.h\"\n \n static int always_advertise(struct repository *r,\n \t\t\t struct strbuf *value)\n@@ -52,6 +53,7 @@ struct protocol_capability {\n static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n \t{ \"ls-refs\", always_advertise, ls_refs },\n+\t{ \"fetch\", always_advertise, upload_pack_v2 },\n };\n \n static void advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 33536254e..202cb782d 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -9,6 +9,7 @@ test_expect_success 'test capability advertisement' '\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs\n+\tfetch\n \t0000\n \tEOF\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex 1e8a9e1ca..c6518a24d 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -18,6 +18,7 @@\n #include \"prio-queue.h\"\n #include \"protocol.h\"\n #include \"upload-pack.h\"\n+#include \"serve.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -1065,3 +1066,283 @@ void upload_pack(struct upload_pack_options *options)\n \t\tcreate_pack_file();\n \t}\n }\n+\n+struct upload_pack_data {\n+\tstruct object_array wants;\n+\tstruct oid_array haves;\n+\n+\tunsigned stateless_rpc : 1;\n+\n+\tunsigned use_thin_pack : 1;\n+\tunsigned use_ofs_delta : 1;\n+\tunsigned no_progress : 1;\n+\tunsigned use_include_tag : 1;\n+\tunsigned done : 1;\n+};\n+\n+#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }\n+\n+static void upload_pack_data_clear(struct upload_pack_data *data)\n+{\n+\tobject_array_clear(&data->wants);\n+\toid_array_clear(&data->haves);\n+}\n+\n+static int parse_want(const char *line)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"want \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tstruct object *o;\n+\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"git upload-pack: protocol error, \"\n+\t\t\t \"expected to get oid, not '%s'\", line);\n+\n+\t\to = parse_object(&oid);\n+\t\tif (!o) {\n+\t\t\tpacket_write_fmt(1,\n+\t\t\t\t\t \"ERR upload-pack: not our ref %s\",\n+\t\t\t\t\t oid_to_hex(&oid));\n+\t\t\tdie(\"git upload-pack: not our ref %s\",\n+\t\t\t oid_to_hex(&oid));\n+\t\t}\n+\n+\t\tif (!(o->flags & WANTED)) {\n+\t\t\to->flags |= WANTED;\n+\t\t\tadd_object_array(o, NULL, &want_obj);\n+\t\t}\n+\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int parse_have(const char *line, struct oid_array *haves)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"have \", &arg)) {\n+\t\tstruct object_id oid;\n+\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"git upload-pack: expected SHA1 object, got '%s'\", arg);\n+\t\toid_array_append(haves, &oid);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void process_args(struct argv_array *args, struct upload_pack_data *data)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < args->argc; i++) {\n+\t\tconst char *arg = args->argv[i];\n+\n+\t\t/* process want */\n+\t\tif (parse_want(arg))\n+\t\t\tcontinue;\n+\t\t/* process have line */\n+\t\tif (parse_have(arg, &data->haves))\n+\t\t\tcontinue;\n+\n+\t\t/* process args like thin-pack */\n+\t\tif (!strcmp(arg, \"thin-pack\")) {\n+\t\t\tuse_thin_pack = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"ofs-delta\")) {\n+\t\t\tuse_ofs_delta = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"no-progress\")) {\n+\t\t\tno_progress = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"include-tag\")) {\n+\t\t\tuse_include_tag = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"done\")) {\n+\t\t\tdata->done = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* ignore unknown lines maybe? */\n+\t\tdie(\"unexpect line: '%s'\", arg);\n+\t}\n+}\n+\n+static void read_haves(struct upload_pack_data *data)\n+{\n+\tstruct packet_reader reader;\n+\tpacket_reader_init(&reader, 0, NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (packet_reader_read(&reader) == PACKET_READ_NORMAL) {\n+\n+\t\tif (parse_have(reader.line, &data->haves))\n+\t\t\tcontinue;\n+\t\tif (!strcmp(reader.line, \"done\")) {\n+\t\t\tdata->done = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\tif (reader.status != PACKET_READ_FLUSH)\n+\t\tdie(\"ERROR\");\n+}\n+\n+static int process_haves(struct oid_array *haves, struct oid_array *common)\n+{\n+\tint i;\n+\n+\t/* Process haves */\n+\tfor (i = 0; i < haves->nr; i++) {\n+\t\tconst struct object_id *oid = &haves->oid[i];\n+\t\tstruct object *o;\n+\t\tint we_knew_they_have = 0;\n+\n+\t\tif (!has_object_file(oid))\n+\t\t\tcontinue;\n+\n+\t\toid_array_append(common, oid);\n+\n+\t\to = parse_object(oid);\n+\t\tif (!o)\n+\t\t\tdie(\"oops (%s)\", oid_to_hex(oid));\n+\t\tif (o->type == OBJ_COMMIT) {\n+\t\t\tstruct commit_list *parents;\n+\t\t\tstruct commit *commit = (struct commit *)o;\n+\t\t\tif (o->flags & THEY_HAVE)\n+\t\t\t\twe_knew_they_have = 1;\n+\t\t\telse\n+\t\t\t\to->flags |= THEY_HAVE;\n+\t\t\tif (!oldest_have || (commit->date < oldest_have))\n+\t\t\t\toldest_have = commit->date;\n+\t\t\tfor (parents = commit->parents;\n+\t\t\t parents;\n+\t\t\t parents = parents->next)\n+\t\t\t\tparents->item->object.flags |= THEY_HAVE;\n+\t\t}\n+\t\tif (!we_knew_they_have)\n+\t\t\tadd_object_array(o, NULL, &have_obj);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int send_acks(struct oid_array *acks, struct strbuf *response)\n+{\n+\tint i;\n+\n+\tpacket_buf_write(response, \"acknowledgments\\n\");\n+\n+\t/* Send Acks */\n+\tif (!acks->nr)\n+\t\tpacket_buf_write(response, \"NAK\\n\");\n+\n+\tfor (i = 0; i < acks->nr; i++) {\n+\t\tpacket_buf_write(response, \"ACK %s\\n\",\n+\t\t\t\t oid_to_hex(&acks->oid[i]));\n+\t}\n+\n+\tif (ok_to_give_up()) {\n+\t\t/* Send Ready */\n+\t\tpacket_buf_write(response, \"ready\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_haves_and_send_acks(struct upload_pack_data *data)\n+{\n+\tstruct oid_array common = OID_ARRAY_INIT;\n+\tstruct strbuf response = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tprocess_haves(&data->haves, &common);\n+\tif (data->done) {\n+\t\tret = 1;\n+\t} else if (send_acks(&common, &response)) {\n+\t\tpacket_buf_delim(&response);\n+\t\tret = 1;\n+\t} else {\n+\t\t/* Add Flush */\n+\t\tpacket_buf_flush(&response);\n+\t\tret = 0;\n+\t}\n+\n+\t/* Send response */\n+\twrite_or_die(1, response.buf, response.len);\n+\tstrbuf_release(&response);\n+\n+\toid_array_clear(&data->haves);\n+\toid_array_clear(&common);\n+\treturn ret;\n+}\n+\n+enum fetch_state {\n+\tFETCH_PROCESS_ARGS = 0,\n+\tFETCH_READ_HAVES,\n+\tFETCH_SEND_ACKS,\n+\tFETCH_SEND_PACK,\n+\tFETCH_DONE,\n+};\n+\n+int upload_pack_v2(struct repository *r, struct argv_array *keys,\n+\t\t struct argv_array *args)\n+{\n+\tenum fetch_state state = FETCH_PROCESS_ARGS;\n+\tstruct upload_pack_data data = UPLOAD_PACK_DATA_INIT;\n+\tuse_sideband = LARGE_PACKET_MAX;\n+\n+\twhile (state != FETCH_DONE) {\n+\t\tswitch (state) {\n+\t\tcase FETCH_PROCESS_ARGS:\n+\t\t\tprocess_args(args, &data);\n+\n+\t\t\tif (!want_obj.nr) {\n+\t\t\t\t/*\n+\t\t\t\t * Request didn't contain any 'want' lines,\n+\t\t\t\t * guess they didn't want anything.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\t} else if (data.haves.nr) {\n+\t\t\t\t/*\n+\t\t\t\t * Request had 'have' lines, so lets ACK them.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_SEND_ACKS;\n+\t\t\t} else {\n+\t\t\t\t/*\n+\t\t\t\t * Request had 'want's but no 'have's so we can\n+\t\t\t\t * immedietly go to construct and send a pack.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_SEND_PACK;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase FETCH_READ_HAVES:\n+\t\t\tread_haves(&data);\n+\t\t\tstate = FETCH_SEND_ACKS;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_ACKS:\n+\t\t\tif (process_haves_and_send_acks(&data))\n+\t\t\t\tstate = FETCH_SEND_PACK;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_PACK:\n+\t\t\tpacket_write_fmt(1, \"packfile\\n\");\n+\t\t\tcreate_pack_file();\n+\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_DONE:\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n+\tupload_pack_data_clear(&data);\n+\treturn 0;\n+}\ndiff --git a/upload-pack.h b/upload-pack.h\nindex a71e4dc7e..6b7890238 100644\n--- a/upload-pack.h\n+++ b/upload-pack.h\n@@ -10,4 +10,9 @@ struct upload_pack_options {\n \n void upload_pack(struct upload_pack_options *options);\n \n+struct repository;\n+struct argv_array;\n+extern int upload_pack_v2(struct repository *r, struct argv_array *keys,\n+\t\t\t struct argv_array *args);\n+\n #endif /* UPLOAD_PACK_H */\n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338587","messageId":"20180207011312.189834-8-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v3 07/35] connect: convert get_remote_heads to use struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-07T01:12:44Z","receivedAt":"2018-02-07T01:15:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to allow for better control flow when protocol_v2 is introduced\nconvert 'get_remote_heads()' to use 'struct packet_reader' to read\npacket lines. This enables a client to be able to peek the first line\nof a server's response (without consuming it) in order to determine the\nprotocol version its speaking and then passing control to the\nappropriate handler.\n\nThis is needed because the initial response from a server speaking\nprotocol_v0 includes the first ref, while subsequent protocol versions\nrespond with a version line. We want to be able to read this first line\nwithout consuming the first ref sent in the protocol_v0 case so that the\nprotocol version the server is speaking can be determined outside of\n'get_remote_heads()' in a future patch.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 174 ++++++++++++++++++++++++++++++++++----------------------------\n 1 file changed, 96 insertions(+), 78 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex c3a014c5b..00e90075c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -48,6 +48,12 @@ int check_ref_type(const struct ref *ref, int flags)\n \n static void die_initial_contact(int unexpected)\n {\n+\t/*\n+\t * A hang-up after seeing some response from the other end\n+\t * means that it is unexpected, as we know the other end is\n+\t * willing to talk to us. A hang-up before seeing any\n+\t * response does not necessarily mean an ACL problem, though.\n+\t */\n \tif (unexpected)\n \t\tdie(_(\"The remote end hung up upon initial contact\"));\n \telse\n@@ -56,6 +62,41 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n+static enum protocol_version discover_version(struct packet_reader *reader)\n+{\n+\tenum protocol_version version = protocol_unknown_version;\n+\n+\t/*\n+\t * Peek the first line of the server's response to\n+\t * determine the protocol version the server is speaking.\n+\t */\n+\tswitch (packet_reader_peek(reader)) {\n+\tcase PACKET_READ_EOF:\n+\t\tdie_initial_contact(0);\n+\tcase PACKET_READ_FLUSH:\n+\tcase PACKET_READ_DELIM:\n+\t\tversion = protocol_v0;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\tversion = determine_protocol_version_client(reader->line);\n+\t\tbreak;\n+\t}\n+\n+\t/* Maybe process capabilities here, at least for v2 */\n+\tswitch (version) {\n+\tcase protocol_v1:\n+\t\t/* Read the peeked version line */\n+\t\tpacket_reader_read(reader);\n+\t\tbreak;\n+\tcase protocol_v0:\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tdie(\"unknown protocol version: '%s'\\n\", reader->line);\n+\t}\n+\n+\treturn version;\n+}\n+\n static void parse_one_symref_info(struct string_list *symref, const char *val, int len)\n {\n \tchar *sym, *target;\n@@ -109,60 +150,21 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n-/*\n- * Read one line of a server's ref advertisement into packet_buffer.\n- */\n-static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n-\t\t\t int *responded)\n+static void process_capabilities(const char *line, int *len)\n {\n-\tint len = packet_read(in, src_buf, src_len,\n-\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\tconst char *arg;\n-\tif (len < 0)\n-\t\tdie_initial_contact(*responded);\n-\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n-\t\tdie(\"remote error: %s\", arg);\n-\n-\t*responded = 1;\n-\n-\treturn len;\n-}\n-\n-#define EXPECTING_PROTOCOL_VERSION 0\n-#define EXPECTING_FIRST_REF 1\n-#define EXPECTING_REF 2\n-#define EXPECTING_SHALLOW 3\n-\n-/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n-static int process_protocol_version(void)\n-{\n-\tswitch (determine_protocol_version_client(packet_buffer)) {\n-\tcase protocol_v1:\n-\t\treturn 1;\n-\tcase protocol_v0:\n-\t\treturn 0;\n-\tdefault:\n-\t\tdie(\"server is speaking an unknown protocol\");\n-\t}\n-}\n-\n-static void process_capabilities(int *len)\n-{\n-\tint nul_location = strlen(packet_buffer);\n+\tint nul_location = strlen(line);\n \tif (nul_location == *len)\n \t\treturn;\n-\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n+\tserver_capabilities = xstrdup(line + nul_location + 1);\n \t*len = nul_location;\n }\n \n-static int process_dummy_ref(void)\n+static int process_dummy_ref(const char *line)\n {\n \tstruct object_id oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(packet_buffer, &oid, &name))\n+\tif (parse_oid_hex(line, &oid, &name))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -171,20 +173,20 @@ static int process_dummy_ref(void)\n \treturn !oidcmp(&null_oid, &oid) && !strcmp(name, \"capabilities^{}\");\n }\n \n-static void check_no_capabilities(int len)\n+static void check_no_capabilities(const char *line, int len)\n {\n-\tif (strlen(packet_buffer) != len)\n+\tif (strlen(line) != len)\n \t\twarning(\"Ignoring capabilities after first line '%s'\",\n-\t\t\tpacket_buffer + strlen(packet_buffer));\n+\t\t\tline + strlen(line));\n }\n \n-static int process_ref(int len, struct ref ***list, unsigned int flags,\n-\t\t struct oid_array *extra_have)\n+static int process_ref(const char *line, int len, struct ref ***list,\n+\t\t unsigned int flags, struct oid_array *extra_have)\n {\n \tstruct object_id old_oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n+\tif (parse_oid_hex(line, &old_oid, &name))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -200,16 +202,17 @@ static int process_ref(int len, struct ref ***list, unsigned int flags,\n \t\t**list = ref;\n \t\t*list = &ref->next;\n \t}\n-\tcheck_no_capabilities(len);\n+\tcheck_no_capabilities(line, len);\n \treturn 1;\n }\n \n-static int process_shallow(int len, struct oid_array *shallow_points)\n+static int process_shallow(const char *line, int len,\n+\t\t\t struct oid_array *shallow_points)\n {\n \tconst char *arg;\n \tstruct object_id old_oid;\n \n-\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\tif (!skip_prefix(line, \"shallow \", &arg))\n \t\treturn 0;\n \n \tif (get_oid_hex(arg, &old_oid))\n@@ -217,10 +220,17 @@ static int process_shallow(int len, struct oid_array *shallow_points)\n \tif (!shallow_points)\n \t\tdie(\"repository on the other end cannot be shallow\");\n \toid_array_append(shallow_points, &old_oid);\n-\tcheck_no_capabilities(len);\n+\tcheck_no_capabilities(line, len);\n \treturn 1;\n }\n \n+enum get_remote_heads_state {\n+\tEXPECTING_FIRST_REF = 0,\n+\tEXPECTING_REF,\n+\tEXPECTING_SHALLOW,\n+\tEXPECTING_DONE,\n+};\n+\n /*\n * Read all the refs from the other end\n */\n@@ -230,47 +240,55 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t struct oid_array *shallow_points)\n {\n \tstruct ref **orig_list = list;\n+\tint len = 0;\n+\tenum get_remote_heads_state state = EXPECTING_FIRST_REF;\n+\tstruct packet_reader reader;\n+\tconst char *arg;\n \n-\t/*\n-\t * A hang-up after seeing some response from the other end\n-\t * means that it is unexpected, as we know the other end is\n-\t * willing to talk to us. A hang-up before seeing any\n-\t * response does not necessarily mean an ACL problem, though.\n-\t */\n-\tint responded = 0;\n-\tint len;\n-\tint state = EXPECTING_PROTOCOL_VERSION;\n+\tpacket_reader_init(&reader, in, src_buf, src_len,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tdiscover_version(&reader);\n \n \t*list = NULL;\n \n-\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\twhile (state != EXPECTING_DONE) {\n+\t\tswitch (packet_reader_read(&reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tdie_initial_contact(1);\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tlen = reader.pktlen;\n+\t\t\tif (len > 4 && skip_prefix(reader.line, \"ERR \", &arg))\n+\t\t\t\tdie(\"remote error: %s\", arg);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tstate = EXPECTING_DONE;\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tdie(\"invalid packet\");\n+\t\t}\n+\n \t\tswitch (state) {\n-\t\tcase EXPECTING_PROTOCOL_VERSION:\n-\t\t\tif (process_protocol_version()) {\n-\t\t\t\tstate = EXPECTING_FIRST_REF;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tstate = EXPECTING_FIRST_REF;\n-\t\t\t/* fallthrough */\n \t\tcase EXPECTING_FIRST_REF:\n-\t\t\tprocess_capabilities(&len);\n-\t\t\tif (process_dummy_ref()) {\n+\t\t\tprocess_capabilities(reader.line, &len);\n+\t\t\tif (process_dummy_ref(reader.line)) {\n \t\t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tstate = EXPECTING_REF;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_REF:\n-\t\t\tif (process_ref(len, &list, flags, extra_have))\n+\t\t\tif (process_ref(reader.line, len, &list, flags, extra_have))\n \t\t\t\tbreak;\n \t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_SHALLOW:\n-\t\t\tif (process_shallow(len, shallow_points))\n+\t\t\tif (process_shallow(reader.line, len, shallow_points))\n \t\t\t\tbreak;\n-\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n-\t\tdefault:\n-\t\t\tdie(\"unexpected state %d\", state);\n+\t\t\tdie(\"protocol error: unexpected '%s'\", reader.line);\n+\t\tcase EXPECTING_DONE:\n+\t\t\tbreak;\n \t\t}\n \t}\n \n-- \n2.16.0.rc1.238.g530d649a79-goog\n\n"},{"id":"338652","messageId":"CAGZ79kauMhgP4Gjm=yVKSDGh0-WMSpNVCJRamfjnANyO5SWXTQ@mail.gmail.com","threadId":"47524","inReplyTo":"20180207011312.189834-23-bmwill@google.com","subject":"Re: [PATCH v3 22/35] upload-pack: support shallow requests","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-07T19:00:37Z","receivedAt":"2018-02-07T19:00:44Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> Add the 'shallow' feature to the protocol version 2 command 'fetch'\n> which indicates that the server supports shallow clients and deepen\n> requets.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> Documentation/technical/protocol-v2.txt | 67 +++++++++++++++-\n> serve.c | 2 +-\n> t/t5701-git-serve.sh | 2 +-\n> upload-pack.c | 138 +++++++++++++++++++++++---------\n> upload-pack.h | 3 +\n> 5 files changed, 173 insertions(+), 39 deletions(-)\n>\n> diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> index 4d5096dae..fedeb6b77 100644\n> --- a/Documentation/technical/protocol-v2.txt\n> +++ b/Documentation/technical/protocol-v2.txt\n> @@ -201,12 +201,42 @@ packet-lines:\n> to its base by position in pack rather than by an oid. That is,\n> they can read OBJ_OFS_DELTA (ake type 6) in a packfile.\n>\n> + shallow \n> + A client must notify the server of all objects for which it only\n\ns/all objects/all commits/ for preciseness\n\n> + has shallow copies of (meaning that it doesn't have the parents\n> + of a commit) by supplying a 'shallow ' line for each such\n> + object so that the serve is aware of the limitations of the\n> + client's history.\n> +\n> + deepen \n> + Request that the fetch/clone should be shallow having a commit depth of\n> + relative to the remote side.\n\nWhat does depth mean? number of commits, or number of edges?\nAre there any special numbers (-1, 0, 1, max int) ?\n\nFrom reading ahead: \"Cannot be used with deepen-since, but\ncan be combined with deepen-relative\" ?\n\n\n> +\n> + deepen-relative\n> + Requests that the semantics of the \"deepen\" command be changed\n> + to indicate that the depth requested is relative to the clients\n> + current shallow boundary, instead of relative to the remote\n> + refs.\n> +\n> + deepen-since \n> + Requests that the shallow clone/fetch should be cut at a\n> + specific time, instead of depth. Internally it's equivalent of\n> + doing \"rev-list --max-age=\". Cannot be used with\n> + \"deepen\".\n> +\n> + deepen-not \n> + Requests that the shallow clone/fetch should be cut at a\n> + specific revision specified by '', instead of a depth.\n> + Internally it's equivalent of doing \"rev-list --not \".\n> + Cannot be used with \"deepen\", but can be used with\n> + \"deepen-since\".\n\nWhat happens if those are given in combination?\n"},{"id":"338978","messageId":"CACsJy8DhLzkxAr7oaBfb22x-bOOPxXxsKKwo5WKJ8p=19BSezg@mail.gmail.com","threadId":"47524","inReplyTo":"CAGZ79kauMhgP4Gjm=yVKSDGh0-WMSpNVCJRamfjnANyO5SWXTQ@mail.gmail.com","subject":"Re: [PATCH v3 22/35] upload-pack: support shallow requests","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-02-10T10:23:45Z","receivedAt":"2018-02-10T10:24:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Feb 8, 2018 at 2:00 AM, Stefan Beller wrote:\n>> +\n>> + deepen-relative\n>> + Requests that the semantics of the \"deepen\" command be changed\n>> + to indicate that the depth requested is relative to the clients\n>> + current shallow boundary, instead of relative to the remote\n>> + refs.\n>> +\n>> + deepen-since \n>> + Requests that the shallow clone/fetch should be cut at a\n>> + specific time, instead of depth. Internally it's equivalent of\n>> + doing \"rev-list --max-age=\". Cannot be used with\n>> + \"deepen\".\n>> +\n>> + deepen-not \n>> + Requests that the shallow clone/fetch should be cut at a\n>> + specific revision specified by '', instead of a depth.\n>> + Internally it's equivalent of doing \"rev-list --not \".\n>> + Cannot be used with \"deepen\", but can be used with\n>> + \"deepen-since\".\n>\n> What happens if those are given in combination?\n\nIt should be described in the old protocol document or I did a bad job\ndocumenting it. Some of these can be combined (I think it's AND logic\nfrom rev-list point of view), with the exception of --depth which does\nnot use rev-list underneath and cannot be combined with the others.\n-- \nDuy\n"},{"id":"339078","messageId":"014a6a12-cf36-35c4-25e0-9e418aec3561@gmail.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"Re: [PATCH v3 00/35] protocol version 2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-12T14:50:40Z","receivedAt":"2018-02-12T14:50:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/6/2018 8:12 PM, Brandon Williams wrote:\n> Changes in v3:\n> * There were some comments about how the protocol should be designed\n> stateless first. I've made this change and instead of having to\n> supply the `stateless-rpc=true` capability to force stateless\n> behavior, the protocol just requires all commands to be stateless.\n> \n> * Added some patches towards the end of the series to force the client\n> to not request to use protocol v2 when pushing (even if configured to\n> use v2). This is to ease the roll-out process of a push command in\n> protocol v2. This way when servers gain the ability to accept\n> pushing in v2 (and they start responding using v2 when requests are\n> sent to the git-receive-pack endpoint) that clients who still don't\n> understand how to push using v2 won't request to use v2 and then die\n> when they recognize that the server does indeed know how to accept a\n> push under v2.\n>\n> * I implemented the `shallow` feature for fetch. This feature\n> encapsulates the existing functionality of all the shallow/deepen\n> capabilities in v0. So now a server can process shallow requests.\n>\n> * Various other small tweaks that I can't remember :)\n>\n> After all of that I think the series is in a pretty good state, baring\n> any more critical reviewing feedback.\n>\n> Thanks!\n>\n> Brandon Williams (35):\n> pkt-line: introduce packet_read_with_status\n> pkt-line: introduce struct packet_reader\n> pkt-line: add delim packet support\n> upload-pack: convert to a builtin\n> upload-pack: factor out processing lines\n> transport: use get_refs_via_connect to get refs\n> connect: convert get_remote_heads to use struct packet_reader\n> connect: discover protocol version outside of get_remote_heads\n> transport: store protocol version\n> protocol: introduce enum protocol_version value protocol_v2\n> test-pkt-line: introduce a packet-line test helper\n> serve: introduce git-serve\n> ls-refs: introduce ls-refs server command\n> connect: request remote refs using v2\n> transport: convert get_refs_list to take a list of ref patterns\n> transport: convert transport_get_remote_refs to take a list of ref\n> patterns\n> ls-remote: pass ref patterns when requesting a remote's refs\n> fetch: pass ref patterns when fetching\n> push: pass ref patterns when pushing\n> upload-pack: introduce fetch server command\n> fetch-pack: perform a fetch using v2\n> upload-pack: support shallow requests\n> fetch-pack: support shallow requests\n> connect: refactor git_connect to only get the protocol version once\n> connect: don't request v2 when pushing\n> transport-helper: remove name parameter\n> transport-helper: refactor process_connect_service\n> transport-helper: introduce stateless-connect\n> pkt-line: add packet_buf_write_len function\n> remote-curl: create copy of the service name\n> remote-curl: store the protocol version the server responded with\n> http: allow providing extra headers for http requests\n> http: don't always add Git-Protocol header\n> remote-curl: implement stateless-connect command\n> remote-curl: don't request v2 when pushing\n>\n> .gitignore | 1 +\n> Documentation/technical/protocol-v2.txt | 338 +++++++++++++++++\n> Makefile | 7 +-\n> builtin.h | 2 +\n> builtin/clone.c | 2 +-\n> builtin/fetch-pack.c | 21 +-\n> builtin/fetch.c | 14 +-\n> builtin/ls-remote.c | 7 +-\n> builtin/receive-pack.c | 6 +\n> builtin/remote.c | 2 +-\n> builtin/send-pack.c | 20 +-\n> builtin/serve.c | 30 ++\n> builtin/upload-pack.c | 74 ++++\n> connect.c | 352 +++++++++++++-----\n> connect.h | 7 +\n> fetch-pack.c | 319 +++++++++++++++-\n> fetch-pack.h | 4 +-\n> git.c | 2 +\n> http.c | 25 +-\n> http.h | 2 +\n> ls-refs.c | 96 +++++\n> ls-refs.h | 9 +\n> pkt-line.c | 149 +++++++-\n> pkt-line.h | 77 ++++\n> protocol.c | 2 +\n> protocol.h | 1 +\n> remote-curl.c | 257 ++++++++++++-\n> remote.h | 9 +-\n> serve.c | 260 +++++++++++++\n> serve.h | 15 +\n> t/helper/test-pkt-line.c | 64 ++++\n> t/t5701-git-serve.sh | 176 +++++++++\n> t/t5702-protocol-v2.sh | 239 ++++++++++++\n> transport-helper.c | 84 +++--\n> transport-internal.h | 4 +-\n> transport.c | 116 ++++--\n> transport.h | 9 +-\n> upload-pack.c | 625 ++++++++++++++++++++++++--------\n> upload-pack.h | 21 ++\n> 39 files changed, 3088 insertions(+), 360 deletions(-)\n> create mode 100644 Documentation/technical/protocol-v2.txt\n> create mode 100644 builtin/serve.c\n> create mode 100644 builtin/upload-pack.c\n> create mode 100644 ls-refs.c\n> create mode 100644 ls-refs.h\n> create mode 100644 serve.c\n> create mode 100644 serve.h\n> create mode 100644 t/helper/test-pkt-line.c\n> create mode 100755 t/t5701-git-serve.sh\n> create mode 100755 t/t5702-protocol-v2.sh\n> create mode 100644 upload-pack.h\n>\n\nI inspected the diff between v2 and v3 and found the changes to be good.\n\nReviewed-By: Derrick Stolee \n\nThanks,\n-Stolee\n\n"},{"id":"339129","messageId":"20180213002554.GA42272@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-2-bmwill@google.com","subject":"Re: [PATCH v3 01/35] pkt-line: introduce packet_read_with_status","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-13T00:25:54Z","receivedAt":"2018-02-13T00:32:27Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> The current pkt-line API encodes the status of a pkt-line read in the\n> length of the read content. An error is indicated with '-1', a flush\n> with '0' (which can be confusing since a return value of '0' can also\n> indicate an empty pkt-line), and a positive integer for the length of\n> the read content otherwise. This doesn't leave much room for allowing\n> the addition of additional special packets in the future.\n>\n> To solve this introduce 'packet_read_with_status()' which reads a packet\n> and returns the status of the read encoded as an 'enum packet_status'\n> type. This allows for easily identifying between special and normal\n> packets as well as errors. It also enables easily adding a new special\n> packet in the future.\n\nMakes sense, thanks. Using an enum return value is less opaque, too.\n\n[...]\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -280,28 +280,33 @@ static int packet_length(const char *linelen)\n> \treturn (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);\n> }\n> \n> -int packet_read(int fd, char **src_buf, size_t *src_len,\n> -\t\tchar *buffer, unsigned size, int options)\n> +enum packet_read_status packet_read_with_status(int fd, char **src_buffer, size_t *src_len,\n> +\t\t\t\t\t\tchar *buffer, unsigned size, int *pktlen,\n> +\t\t\t\t\t\tint options)\n\nThis function definition straddles two worlds: it is line-wrapped as\nthough there are a limited number of columns, but it goes far past 80\ncolumns.\n\nCan \"make style\" or a similar tool take care of rewrapping it?\n\n\n> {\n> -\tint len, ret;\n> +\tint len;\n> \tchar linelen[4];\n> \n> -\tret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);\n> -\tif (ret < 0)\n> -\t\treturn ret;\n> +\tif (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)\n> +\t\treturn PACKET_READ_EOF;\n> +\n\nEOF is indeed the only error that get_packet_data can return.\n\nCould be worth a doc comment on get_packet_data to make that clearer.\nIt's not too important since it's static, though.\n\n> \tlen = packet_length(linelen);\n> -\tif (len < 0)\n> +\n> +\tif (len < 0) {\n> \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n> -\tif (!len) {\n> +\t} else if (!len) {\n> \t\tpacket_trace(\"0000\", 4, 0);\n> -\t\treturn 0;\n> +\t\treturn PACKET_READ_FLUSH;\n\nThe advertised change. Makes sense.\n\n[...]\n> -\tif (len >= size)\n> +\tif ((unsigned)len >= size)\n> \t\tdie(\"protocol error: bad line length %d\", len);\n\nThe comparison is safe since we just checked that len >= 0.\n\nIs there some static analysis that can make this kind of operation\neasier?\n\n[...]\n> @@ -309,7 +314,31 @@ int packet_read(int fd, char **src_buf, size_t *src_len,\n> \n> \tbuffer[len] = 0;\n> \tpacket_trace(buffer, len, 0);\n> -\treturn len;\n> +\t*pktlen = len;\n> +\treturn PACKET_READ_NORMAL;\n> +}\n> +\n> +int packet_read(int fd, char **src_buffer, size_t *src_len,\n> +\t\tchar *buffer, unsigned size, int options)\n> +{\n> +\tenum packet_read_status status;\n> +\tint pktlen;\n> +\n> +\tstatus = packet_read_with_status(fd, src_buffer, src_len,\n> +\t\t\t\t\t buffer, size, &pktlen,\n> +\t\t\t\t\t options);\n> +\tswitch (status) {\n> +\tcase PACKET_READ_EOF:\n> +\t\tpktlen = -1;\n> +\t\tbreak;\n> +\tcase PACKET_READ_NORMAL:\n> +\t\tbreak;\n> +\tcase PACKET_READ_FLUSH:\n> +\t\tpktlen = 0;\n> +\t\tbreak;\n> +\t}\n> +\n> +\treturn pktlen;\n\nnit: can simplify by avoiding the status temporary:\n\n\tint pktlen;\n\n\tswitch (packet_read_with_status(...)) {\n\tcase PACKET_READ_EOF:\n\t\treturn -1;\n\tcase PACKET_READ_FLUSH:\n\t\treturn 0;\n\tcase PACKET_READ_NORMAL:\n\t\treturn pktlen;\n\t}\n\nAs a bonus, that lets static analyzers check that the cases are\nexhaustive. (On the other hand, C doesn't guarantee that an enum can\nonly have the values listed as enumerators. Did we end up figuring\nout a way to handle that, beyond always including a 'default: BUG()'?)\n\n> --- a/pkt-line.h\n> +++ b/pkt-line.h\n> @@ -65,6 +65,21 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n> int packet_read(int fd, char **src_buffer, size_t *src_len, char\n> \t\t*buffer, unsigned size, int options);\n> \n> +/*\n> + * Read a packetized line into a buffer like the 'packet_read()' function but\n> + * returns an 'enum packet_read_status' which indicates the status of the read.\n> + * The number of bytes read will be assigined to *pktlen if the status of the\n> + * read was 'PACKET_READ_NORMAL'.\n> + */\n> +enum packet_read_status {\n> +\tPACKET_READ_EOF = -1,\n> +\tPACKET_READ_NORMAL,\n> +\tPACKET_READ_FLUSH,\n> +};\n\nnit: do any callers treat the return value as a number? It would be\nless magical if the numbering were left to the compiler (0, 1, 2).\n\nThanks,\nJonathan\n"},{"id":"339132","messageId":"20180213004937.GB42272@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-3-bmwill@google.com","subject":"Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-13T00:49:37Z","receivedAt":"2018-02-13T00:49:47Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> Subject: pkt-line: introduce struct packet_reader\n\nnit: this subject line doesn't describe what the purpose/intent behind\nthe patch is. Maybe something like\n\n\tpkt-line: allow peeking at a packet line without consuming it\n\nwould make it clearer.\n\n> Sometimes it is advantageous to be able to peek the next packet line\n> without consuming it (e.g. to be able to determine the protocol version\n> a server is speaking). In order to do that introduce 'struct\n> packet_reader' which is an abstraction around the normal packet reading\n> logic. This enables a caller to be able to peek a single line at a time\n> using 'packet_reader_peek()' and having a caller consume a line by\n> calling 'packet_reader_read()'.\n\nMakes sense. The packet_reader owns a buffer to support the peek\noperation and make buffer reuse a little easier.\n\n[...]\n> --- a/pkt-line.h\n> +++ b/pkt-line.h\n> @@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n> */\n> ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);\n> \n> +struct packet_reader {\n> +\t/* source file descriptor */\n> +\tint fd;\n> +\n> +\t/* source buffer and its size */\n> +\tchar *src_buffer;\n> +\tsize_t src_len;\n\nCan or should this be a strbuf?\n\n> +\n> +\t/* buffer that pkt-lines are read into and its size */\n> +\tchar *buffer;\n> +\tunsigned buffer_size;\n\nLikewise.\n\n> +\n> +\t/* options to be used during reads */\n> +\tint options;\n\nWhat option values are possible?\n\n> +\n> +\t/* status of the last read */\n> +\tenum packet_read_status status;\n\nThis reminds me of FILE*'s status value, ferror, etc. I'm mildly\nnervous about it --- it encourages a style of error handling where you\nignore errors from an individual operation and hope the recorded\nstatus later has the most relevant error.\n\nI think it is being used to support peek --- you need to record the\nerror to reply it. Is that right? I wonder if it would make sense to\nstructure as\n\n\t\tstruct packet_read_result last_line_read;\n\t};\n\n\tstruct packet_read_result {\n\t\tenum packet_read_status status;\n\n\t\tconst char *line;\n\t\tint len;\n\t};\n\nWhat you have here also seems fine. I think what would help most\nfor readability is if the comment mentioned the purpose --- e.g.\n\n\t/* status of the last read, to support peeking */\n\nOr if the contract were tied to the purpose:\n\n\t/* status of the last read, only valid if line_peeked is true */\n\n[...]\n> +/*\n> + * Initialize a 'struct packet_reader' object which is an\n> + * abstraction around the 'packet_read_with_status()' function.\n> + */\n> +extern void packet_reader_init(struct packet_reader *reader, int fd,\n> +\t\t\t char *src_buffer, size_t src_len,\n> +\t\t\t int options);\n\nThis comment doesn't describe how I should use the function. Is the\nintent to point the reader to packet_read_with_status for more details\nabout the arguments?\n\nCan src_buffer be a const char *?\n\n[...]\n> +/*\n> + * Perform a packet read and return the status of the read.\n\nnit: s/Perform a packet read/Read one pkt-line/\n\n> + * The values of 'pktlen' and 'line' are updated based on the status of the\n> + * read as follows:\n> + *\n> + * PACKET_READ_ERROR: 'pktlen' is set to '-1' and 'line' is set to NULL\n> + * PACKET_READ_NORMAL: 'pktlen' is set to the number of bytes read\n> + *\t\t 'line' is set to point at the read line\n> + * PACKET_READ_FLUSH: 'pktlen' is set to '0' and 'line' is set to NULL\n> + */\n> +extern enum packet_read_status packet_reader_read(struct packet_reader *reader);\n\nThis is reasonable. As described above an alternative would be\npossible to have a separate packet_read_result output parameter but\nthe interface described here looks pretty easy/pleasant to use.\n\n> +\n> +/*\n> + * Peek the next packet line without consuming it and return the status.\n\nnit: s/Peek/Peek at/, or s/Peek/Read/\n\n> + * The next call to 'packet_reader_read()' will perform a read of the same line\n> + * that was peeked, consuming the line.\n\nnit: s/peeked/peeked at/\n\n> + *\n> + * Peeking multiple times without calling 'packet_reader_read()' will return\n> + * the same result.\n> + */\n> +extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);\n\nNice.\n\n[...]\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -406,3 +406,62 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)\n> \t}\n> \treturn sb_out->len - orig_len;\n> }\n> +\n> +/* Packet Reader Functions */\n> +void packet_reader_init(struct packet_reader *reader, int fd,\n> +\t\t\tchar *src_buffer, size_t src_len,\n> +\t\t\tint options)\n\nThis comment looks like it's attached to packet_reader_init, but it's\nmeant to be attached to the whole collection. It's possible that this\ntitle-above-multiple-functions won't be maintained, but that's okay.\n\n> +{\n> +\tmemset(reader, 0, sizeof(*reader));\n> +\n> +\treader->fd = fd;\n> +\treader->src_buffer = src_buffer;\n> +\treader->src_len = src_len;\n> +\treader->buffer = packet_buffer;\n> +\treader->buffer_size = sizeof(packet_buffer);\n\nLooks like this is very non-reentrant. Can the doc comment warn about\nthat? Or even better, can it be made reentrant by owning its own\nstrbuf?\n\n> +\treader->options = options;\n> +}\n> +\n> +enum packet_read_status packet_reader_read(struct packet_reader *reader)\n> +{\n> +\tif (reader->line_peeked) {\n> +\t\treader->line_peeked = 0;\n> +\t\treturn reader->status;\n> +\t}\n\nNice.\n\n> +\n> +\treader->status = packet_read_with_status(reader->fd,\n> +\t\t\t\t\t\t &reader->src_buffer,\n> +\t\t\t\t\t\t &reader->src_len,\n> +\t\t\t\t\t\t reader->buffer,\n> +\t\t\t\t\t\t reader->buffer_size,\n> +\t\t\t\t\t\t &reader->pktlen,\n> +\t\t\t\t\t\t reader->options);\n> +\n> +\tswitch (reader->status) {\n> +\tcase PACKET_READ_EOF:\n> +\t\treader->pktlen = -1;\n> +\t\treader->line = NULL;\n> +\t\tbreak;\n> +\tcase PACKET_READ_NORMAL:\n> +\t\treader->line = reader->buffer;\n> +\t\tbreak;\n> +\tcase PACKET_READ_FLUSH:\n> +\t\treader->pktlen = 0;\n> +\t\treader->line = NULL;\n> +\t\tbreak;\n> +\t}\n> +\n> +\treturn reader->status;\n> +}\n> +\n> +enum packet_read_status packet_reader_peek(struct packet_reader *reader)\n> +{\n> +\t/* Only allow peeking a single line */\n\nnit: s/peeking at/\n\n> +\tif (reader->line_peeked)\n> +\t\treturn reader->status;\n> +\n> +\t/* Peek a line by reading it and setting peeked flag */\n\nnit: s/Peek/Peek at/\n\n> +\tpacket_reader_read(reader);\n> +\treader->line_peeked = 1;\n> +\treturn reader->status;\n> +}\n\nThanks for a pleasant read.\n\nJonathan\n"},{"id":"339203","messageId":"20180213170645.GC209601@google.com","threadId":"47524","inReplyTo":"CAGZ79kauMhgP4Gjm=yVKSDGh0-WMSpNVCJRamfjnANyO5SWXTQ@mail.gmail.com","subject":"Re: [PATCH v3 22/35] upload-pack: support shallow requests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-13T17:06:45Z","receivedAt":"2018-02-13T17:06:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/07, Stefan Beller wrote:\n> On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> > Add the 'shallow' feature to the protocol version 2 command 'fetch'\n> > which indicates that the server supports shallow clients and deepen\n> > requets.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > Documentation/technical/protocol-v2.txt | 67 +++++++++++++++-\n> > serve.c | 2 +-\n> > t/t5701-git-serve.sh | 2 +-\n> > upload-pack.c | 138 +++++++++++++++++++++++---------\n> > upload-pack.h | 3 +\n> > 5 files changed, 173 insertions(+), 39 deletions(-)\n> >\n> > diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\n> > index 4d5096dae..fedeb6b77 100644\n> > --- a/Documentation/technical/protocol-v2.txt\n> > +++ b/Documentation/technical/protocol-v2.txt\n> > @@ -201,12 +201,42 @@ packet-lines:\n> > to its base by position in pack rather than by an oid. That is,\n> > they can read OBJ_OFS_DELTA (ake type 6) in a packfile.\n> >\n> > + shallow \n> > + A client must notify the server of all objects for which it only\n> \n> s/all objects/all commits/ for preciseness\n> \n> > + has shallow copies of (meaning that it doesn't have the parents\n> > + of a commit) by supplying a 'shallow ' line for each such\n> > + object so that the serve is aware of the limitations of the\n> > + client's history.\n> > +\n> > + deepen \n> > + Request that the fetch/clone should be shallow having a commit depth of\n> > + relative to the remote side.\n> \n> What does depth mean? number of commits, or number of edges?\n> Are there any special numbers (-1, 0, 1, max int) ?\n> \n> From reading ahead: \"Cannot be used with deepen-since, but\n> can be combined with deepen-relative\" ?\n\nIt just uses the current logic, which has no documentation on any of\nthat so...I'm not really sure?\n\n> \n> \n> > +\n> > + deepen-relative\n> > + Requests that the semantics of the \"deepen\" command be changed\n> > + to indicate that the depth requested is relative to the clients\n> > + current shallow boundary, instead of relative to the remote\n> > + refs.\n> > +\n> > + deepen-since \n> > + Requests that the shallow clone/fetch should be cut at a\n> > + specific time, instead of depth. Internally it's equivalent of\n> > + doing \"rev-list --max-age=\". Cannot be used with\n> > + \"deepen\".\n> > +\n> > + deepen-not \n> > + Requests that the shallow clone/fetch should be cut at a\n> > + specific revision specified by '', instead of a depth.\n> > + Internally it's equivalent of doing \"rev-list --not \".\n> > + Cannot be used with \"deepen\", but can be used with\n> > + \"deepen-since\".\n> \n> What happens if those are given in combination?\n\nShould act as an AND, it uses the old logic and there isn't very much\ndocumentation on that...\n\n-- \nBrandon Williams\n"},{"id":"339834","messageId":"20180221200141.GA127348@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"Re: [PATCH v3 00/35] protocol version 2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-21T20:01:41Z","receivedAt":"2018-02-21T20:01:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/06, Brandon Williams wrote:\n> Changes in v3:\n> * There were some comments about how the protocol should be designed\n> stateless first. I've made this change and instead of having to\n> supply the `stateless-rpc=true` capability to force stateless\n> behavior, the protocol just requires all commands to be stateless.\n> \n> * Added some patches towards the end of the series to force the client\n> to not request to use protocol v2 when pushing (even if configured to\n> use v2). This is to ease the roll-out process of a push command in\n> protocol v2. This way when servers gain the ability to accept\n> pushing in v2 (and they start responding using v2 when requests are\n> sent to the git-receive-pack endpoint) that clients who still don't\n> understand how to push using v2 won't request to use v2 and then die\n> when they recognize that the server does indeed know how to accept a\n> push under v2.\n> \n> * I implemented the `shallow` feature for fetch. This feature\n> encapsulates the existing functionality of all the shallow/deepen\n> capabilities in v0. So now a server can process shallow requests.\n> \n> * Various other small tweaks that I can't remember :)\n> \n> After all of that I think the series is in a pretty good state, baring\n> any more critical reviewing feedback.\n> \n> Thanks!\n\nI'm hoping to get some more in depth review before I do any more\nre-rolls, but for those interested I will need to do a re-roll to\neliminate the prelude from the http transport. This is the prelude\nwhich includes the service line followed by any number of packet lines\nculminating in a flush-pkt like so:\n\n # service=git-upload-pack\n some\n other\n optional\n lines\n 0000\n\nWith this eliminated all transports will be exactly the same, the only\ndifference will be how the protocol is tunneled.\n\n> \n> Brandon Williams (35):\n> pkt-line: introduce packet_read_with_status\n> pkt-line: introduce struct packet_reader\n> pkt-line: add delim packet support\n> upload-pack: convert to a builtin\n> upload-pack: factor out processing lines\n> transport: use get_refs_via_connect to get refs\n> connect: convert get_remote_heads to use struct packet_reader\n> connect: discover protocol version outside of get_remote_heads\n> transport: store protocol version\n> protocol: introduce enum protocol_version value protocol_v2\n> test-pkt-line: introduce a packet-line test helper\n> serve: introduce git-serve\n> ls-refs: introduce ls-refs server command\n> connect: request remote refs using v2\n> transport: convert get_refs_list to take a list of ref patterns\n> transport: convert transport_get_remote_refs to take a list of ref\n> patterns\n> ls-remote: pass ref patterns when requesting a remote's refs\n> fetch: pass ref patterns when fetching\n> push: pass ref patterns when pushing\n> upload-pack: introduce fetch server command\n> fetch-pack: perform a fetch using v2\n> upload-pack: support shallow requests\n> fetch-pack: support shallow requests\n> connect: refactor git_connect to only get the protocol version once\n> connect: don't request v2 when pushing\n> transport-helper: remove name parameter\n> transport-helper: refactor process_connect_service\n> transport-helper: introduce stateless-connect\n> pkt-line: add packet_buf_write_len function\n> remote-curl: create copy of the service name\n> remote-curl: store the protocol version the server responded with\n> http: allow providing extra headers for http requests\n> http: don't always add Git-Protocol header\n> remote-curl: implement stateless-connect command\n> remote-curl: don't request v2 when pushing\n> \n> .gitignore | 1 +\n> Documentation/technical/protocol-v2.txt | 338 +++++++++++++++++\n> Makefile | 7 +-\n> builtin.h | 2 +\n> builtin/clone.c | 2 +-\n> builtin/fetch-pack.c | 21 +-\n> builtin/fetch.c | 14 +-\n> builtin/ls-remote.c | 7 +-\n> builtin/receive-pack.c | 6 +\n> builtin/remote.c | 2 +-\n> builtin/send-pack.c | 20 +-\n> builtin/serve.c | 30 ++\n> builtin/upload-pack.c | 74 ++++\n> connect.c | 352 +++++++++++++-----\n> connect.h | 7 +\n> fetch-pack.c | 319 +++++++++++++++-\n> fetch-pack.h | 4 +-\n> git.c | 2 +\n> http.c | 25 +-\n> http.h | 2 +\n> ls-refs.c | 96 +++++\n> ls-refs.h | 9 +\n> pkt-line.c | 149 +++++++-\n> pkt-line.h | 77 ++++\n> protocol.c | 2 +\n> protocol.h | 1 +\n> remote-curl.c | 257 ++++++++++++-\n> remote.h | 9 +-\n> serve.c | 260 +++++++++++++\n> serve.h | 15 +\n> t/helper/test-pkt-line.c | 64 ++++\n> t/t5701-git-serve.sh | 176 +++++++++\n> t/t5702-protocol-v2.sh | 239 ++++++++++++\n> transport-helper.c | 84 +++--\n> transport-internal.h | 4 +-\n> transport.c | 116 ++++--\n> transport.h | 9 +-\n> upload-pack.c | 625 ++++++++++++++++++++++++--------\n> upload-pack.h | 21 ++\n> 39 files changed, 3088 insertions(+), 360 deletions(-)\n> create mode 100644 Documentation/technical/protocol-v2.txt\n> create mode 100644 builtin/serve.c\n> create mode 100644 builtin/upload-pack.c\n> create mode 100644 ls-refs.c\n> create mode 100644 ls-refs.h\n> create mode 100644 serve.c\n> create mode 100644 serve.h\n> create mode 100644 t/helper/test-pkt-line.c\n> create mode 100755 t/t5701-git-serve.sh\n> create mode 100755 t/t5702-protocol-v2.sh\n> create mode 100644 upload-pack.h\n> \n> -- \n> 2.16.0.rc1.238.g530d649a79-goog\n> \n\n-- \nBrandon Williams\n"},{"id":"339840","messageId":"20180221134422.2386e1aca39fe673235590e7@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-5-bmwill@google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-21T21:44:22Z","receivedAt":"2018-02-21T21:44:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:12:41 -0800\nBrandon Williams wrote:\n\n> In order to allow for code sharing with the server-side of fetch in\n> protocol-v2 convert upload-pack to be a builtin.\n> \n> Signed-off-by: Brandon Williams \n\nAs Stefan mentioned in [1], also mention in the commit message that this\nmeans that the \"git-upload-pack\" invocation gains additional\ncapabilities (for example, invoking a pager for --help).\n\nHaving said that, the main purpose of this patch seems to be to libify\nupload-pack, and the move to builtin is just a way of putting the\nprogram somewhere - we could have easily renamed upload-pack.c and\ncreated a new upload-pack.c containing the main(), preserving the\nnon-builtin-ness of upload-pack, while still gaining the benefits of\nlibifying upload-pack.\n\nIf the community does want to make upload-pack a builtin, I would write\nthe commit message this way:\n\n upload-pack: libify\n\n Libify upload-pack. The main() function is moved to\n builtin/upload-pack.c, thus making upload-pack a builtin. Note that\n this means that \"git-upload-pack\" gains functionality such as the\n ability to invoke a pager when passed \"--help\".\n\nAnd if not:\n\n upload-pack: libify\n\n Libify upload-pack by moving most of the functionality in\n upload-pack.c into a file upload-pack-lib.c (or some other name),\n to be used in subsequent patches.\n\n[1] https://public-inbox.org/git/CAGZ79kb2=uU0_K8wr27gNdNX-T+P+7gVdgc5EBdYc3zBobsR8w@mail.gmail.com/\n\n> -static void upload_pack(void)\n> -{\n> -\tstruct string_list symref = STRING_LIST_INIT_DUP;\n> -\n> -\thead_ref_namespaced(find_symref, &symref);\n> -\n> -\tif (advertise_refs || !stateless_rpc) {\n> -\t\treset_timeout();\n> -\t\thead_ref_namespaced(send_ref, &symref);\n> -\t\tfor_each_namespaced_ref(send_ref, &symref);\n> -\t\tadvertise_shallow_grafts(1);\n> -\t\tpacket_flush(1);\n> -\t} else {\n> -\t\thead_ref_namespaced(check_ref, NULL);\n> -\t\tfor_each_namespaced_ref(check_ref, NULL);\n> -\t}\n> -\tstring_list_clear(&symref, 1);\n> -\tif (advertise_refs)\n> -\t\treturn;\n> -\n> -\treceive_needs();\n> -\tif (want_obj.nr) {\n> -\t\tget_common_commits();\n> -\t\tcreate_pack_file();\n> -\t}\n> -}\n\nI see that this function had to be moved to the bottom because it now\nalso needs to make use of functions like upload_pack_config() - that's\nfine.\n\n> +struct upload_pack_options {\n> +\tint stateless_rpc;\n> +\tint advertise_refs;\n> +\tunsigned int timeout;\n> +\tint daemon_mode;\n> +};\n\nI would have expected \"unsigned stateless_rpc : 1\" etc., but I see that\nthis makes it easier to use with OPT_BOOL (which needs us to pass it a\npointer-to-int).\n\nAs for what existing code does, files like fetch-pack and diff use\n\"unsigned : 1\", but they also process arguments without OPT_, so I don't\nthink they are relevant.\n\nI think that we should decide if we're going to prefer \"unsigned : 1\" or\n\"int\" for flags in new code. Personally, I prefer \"unsigned : 1\"\n(despite the slight inconvenience in that argument parsers will need to\ndeclare their own temporary \"int\" and then assign its contents to the\noptions struct) because of the stronger type, but I'm OK either way.\nWhatever the decision, I don't think it needs to block the review of\nthis patch set.\n"},{"id":"339841","messageId":"20180221214755.GA10327@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180103203956.GA44365@google.com","subject":"Re: [PATCH 04/26] upload-pack: convert to a builtin","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-21T21:47:55Z","receivedAt":"2018-02-21T21:48:45Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n> On 01/03, Stefan Beller wrote:\n> > On Tue, Jan 2, 2018 at 4:18 PM, Brandon Williams wrote:\n\n>>> In order to allow for code sharing with the server-side of fetch in\n>>> protocol-v2 convert upload-pack to be a builtin.\n>>\n>> What is the security aspect of this patch?\n>>\n>> By making upload-pack builtin, it gains additional abilities,\n>> such as answers to '-h' or '--help' (which would start a pager).\n>> Is there an easy way to sooth my concerns? (best put into the\n>> commit message)\n>\n> receive-pack is already a builtin, so theres that.\n\n*nod*\n\nSince v2.4.12~1^2 (shell: disallow repo names beginning with dash,\n2017-04-29), git-shell refuses to pass --help to upload-pack, limiting\nthe security impact in configurations that use git-shell (e.g.\ngitolite installations).\n\nIf you're not using git-shell, then hopefully you have some other form\nof filtering preventing arbitrary options being passed to\ngit-upload-pack. If you don't, then you're in trouble, for the\nreasons described in that commit.\n\nSince some installations may be allowing access to git-upload-pack\n(for read-only access) without allowing access to git-receive-pack,\nthis does increase the chance of attack. On the other hand, I suspect\nthe maintainability benefit is worth it.\n\nFor defense in depth, it would be comforting if the git wrapper had\nsome understanding of \"don't support --help in handle_builtin when\ninvoked as a dashed command\". That is, I don't expect that anyone has\nbeen relying on\n\n\tgit-add --help\n\nacting like\n\n\tgit help add\n\ninstead of printing the usage message from\n\n\tgit add -h\n\nIt's a little fussy because today we rewrite \"git add --help\" to\n\"git-add --help\" before rewriting it to \"git help add\"; we'd have to\nskip that middle hop for this to work.\n\nI don't think that has to block this patch or series, though --- it's\njust a separate thought about hardening.\n\nCc-ing Sitaram Chamarty since he tends to be wiser about this kind of\nthing than I am.\n\nWhat do you think?\n\nThanks,\nJonathan\n"},{"id":"339843","messageId":"20180221141135.c78d72b422f985aae5375bca@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-9-bmwill@google.com","subject":"Re: [PATCH v3 08/35] connect: discover protocol version outside of get_remote_heads","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-21T22:11:35Z","receivedAt":"2018-02-21T22:11:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:12:45 -0800\nBrandon Williams wrote:\n\n> -\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n> +\n> +\tpacket_reader_init(&reader, fd[0], NULL, 0,\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> +\n> +\tswitch (discover_version(&reader)) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n\nThis inlining is repeated a few times, which raises the question: if the\nintention is to keep the v0/1 logic separately from v2, why not have a\nsingle function that wraps them all? Looking at the end result (after\nall the patches in this patch set are applied), it seems that the v2\nversion does not have extra_have or shallow parameters, which is a good\nenough reason for me (I don't think functions that take in many\narguments and then selectively use them is a good idea). I think that\nother reviewers will have this question too, so maybe discuss this in\nthe commit message.\n\n> diff --git a/remote.h b/remote.h\n> index 1f6611be2..2016461df 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n> void free_refs(struct ref *ref);\n> \n> struct oid_array;\n> -extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> +struct packet_reader;\n> +extern struct ref **get_remote_heads(struct packet_reader *reader,\n> \t\t\t\t struct ref **list, unsigned int flags,\n> \t\t\t\t struct oid_array *extra_have,\n> -\t\t\t\t struct oid_array *shallow);\n> +\t\t\t\t struct oid_array *shallow_points);\n\nThis change probably does not belong in this patch, especially since\nremote.c is unchanged.\n"},{"id":"339850","messageId":"20180221144515.d180a26627a33168b19e0a4f@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-13-bmwill@google.com","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-21T22:45:15Z","receivedAt":"2018-02-21T22:45:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:12:49 -0800\nBrandon Williams wrote:\n\n> .gitignore | 1 +\n> Documentation/technical/protocol-v2.txt | 114 +++++++++++++++\n> Makefile | 2 +\n> builtin.h | 1 +\n> builtin/serve.c | 30 ++++\n> git.c | 1 +\n> serve.c | 250 ++++++++++++++++++++++++++++++++\n> serve.h | 15 ++\n> t/t5701-git-serve.sh | 60 ++++++++\n> 9 files changed, 474 insertions(+)\n> create mode 100644 Documentation/technical/protocol-v2.txt\n> create mode 100644 builtin/serve.c\n> create mode 100644 serve.c\n> create mode 100644 serve.h\n> create mode 100755 t/t5701-git-serve.sh\n\nAs someone who is implementing the server side of protocol V2 in JGit, I\nnow have a bit more insight into this :-)\n\nFirst of all, I used to not have a strong opinion on the existence of a\nnew endpoint, but now I think that it's better to *not* have git-serve.\nAs it is, as far as I can tell, upload-pack also needs to support (and\ndoes support, as of the end of this patch set) protocol v2 anyway, so it\nmight be better to merely upgrade upload-pack.\n\n> +A client then responds to select the command it wants with any particular\n> +capabilities or arguments. There is then an optional section where the\n> +client can provide any command specific parameters or queries.\n> +\n> + command-request = command\n> +\t\t capability-list\n> +\t\t (command-args)\n\nIf you are stating that this is optional, write \"*1command-args\". (RFC\n5234 also supports square brackets, but \"*1\" is already used in\npack-protocol.txt and http-protocol.txt.)\n\n> +\t\t flush-pkt\n> + command = PKT-LINE(\"command=\" key LF)\n> + command-args = delim-pkt\n> +\t\t *arg\n> + arg = 1*CHAR\n\narg should be wrapped in PKT-LINE, I think, and terminated by an LF.\n"},{"id":"339853","messageId":"20180221145411.35b2ea84747518a499276bdd@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-15-bmwill@google.com","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-21T22:54:11Z","receivedAt":"2018-02-21T22:54:19Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:12:51 -0800\nBrandon Williams wrote:\n\n> +extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> +\t\t\t\t struct ref **list, int for_push,\n> +\t\t\t\t const struct argv_array *ref_patterns);\n\nI haven't looked at the rest of this patch in detail, but the type of\nref_patterns is probably better as struct string_list, since this is not\na true argument array (e.g. with flags starting with --). Same comment\nfor the next few patches that deal with ref patterns.\n"},{"id":"339854","messageId":"20180221145639.c6cf2409ce2120109bdd169f@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-16-bmwill@google.com","subject":"Re: [PATCH v3 15/35] transport: convert get_refs_list to take a list of ref patterns","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-21T22:56:39Z","receivedAt":"2018-02-21T22:56:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:12:52 -0800\nBrandon Williams wrote:\n\n> @@ -21,7 +22,8 @@ struct transport_vtable {\n> \t * the ref without a huge amount of effort, it should store it\n> \t * in the ref's old_sha1 field; otherwise it should be all 0.\n> \t **/\n> -\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push);\n> +\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n> +\t\t\t\t const struct argv_array *ref_patterns);\n\nAlso mention in the documentation that this function is allowed to\nreturn refs that do not match the ref patterns.\n"},{"id":"339855","messageId":"20180221145804.f483fb21559f71618812404d@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-17-bmwill@google.com","subject":"Re: [PATCH v3 16/35] transport: convert transport_get_remote_refs to take a list of ref patterns","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-21T22:58:04Z","receivedAt":"2018-02-21T22:58:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:12:53 -0800\nBrandon Williams wrote:\n\n> -const struct ref *transport_get_remote_refs(struct transport *transport)\n> +const struct ref *transport_get_remote_refs(struct transport *transport,\n> +\t\t\t\t\t const struct argv_array *ref_patterns)\n> {\n> \tif (!transport->got_remote_refs) {\n> -\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n> +\t\ttransport->remote_refs =\n> +\t\t\ttransport->vtable->get_refs_list(transport, 0,\n> +\t\t\t\t\t\t\t ref_patterns);\n> \t\ttransport->got_remote_refs = 1;\n> \t}\n\nShould we do our own client-side filtering if the server side cannot do\nit for us (because it doesn't support protocol v2)? Either way, this\ndecision should be mentioned in the commit message.\n"},{"id":"339864","messageId":"xmqqefldykyx.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180221214755.GA10327@aiede.svl.corp.google.com","subject":"Re: [PATCH 04/26] upload-pack: convert to a builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-21T23:35:50Z","receivedAt":"2018-02-21T23:35:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> For defense in depth, it would be comforting if the git wrapper had\n> some understanding of \"don't support --help in handle_builtin when\n> invoked as a dashed command\". That is, I don't expect that anyone has\n> been relying on\n>\n> \tgit-add --help\n>\n> acting like\n>\n> \tgit help add\n>\n> instead of printing the usage message from\n>\n> \tgit add -h\n\nSounds like a neat trick.\n\n> It's a little fussy because today we rewrite \"git add --help\" to\n> \"git-add --help\" before rewriting it to \"git help add\"; we'd have to\n> skip that middle hop for this to work.\n\nI do not quite get this part. \"git add --help\" goes through run_argv()\nand then to handle_builtin() which is what does this \"git help add\"\nswapping.\n\n\"git-add --help\" does get thrown into the same codepath by\npretending as if we got \"add --help\" as an argument to \"git\"\ncommand, and that happens without going through run_argv(),\nso presumably we can add another perameter to handle_builtin()\nso that the callee can tell these two invocation sites apart, no?\n\n> I don't think that has to block this patch or series, though --- it's\n> just a separate thought about hardening.\n\nYeah, I agree with this assessment.\n"},{"id":"339866","messageId":"20180221154622.f35e69f0d8c6bb54e40a299a@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-21-bmwill@google.com","subject":"Re: [PATCH v3 20/35] upload-pack: introduce fetch server command","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-21T23:46:22Z","receivedAt":"2018-02-21T23:46:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:12:57 -0800\nBrandon Williams wrote:\n\n> + want \n> +\tIndicates to the server an object which the client wants to\n> +\tretrieve.\n\nMention that the client can \"want\" anything even if not advertised by\nthe server (like uploadpack.allowanysha1inwant).\n\n> + output = *section\n> + section = (acknowledgments | packfile)\n> +\t (flush-pkt | delim-pkt)\n> +\n> + acknowledgments = PKT-LINE(\"acknowledgments\" LF)\n> +\t\t *(ready | nak | ack)\n\nCan this part be described more precisely in the BNF section? I see that\nyou describe later that there can be multiple ACKs or one NAK (but not\nboth), and \"ready\" can be sent regardless of whether ACKs or a NAK is\nsent.\n\n> + ready = PKT-LINE(\"ready\" LF)\n> + nak = PKT-LINE(\"NAK\" LF)\n> + ack = PKT-LINE(\"ACK\" SP obj-id LF)\n> +\n> + packfile = PKT-LINE(\"packfile\" LF)\n> +\t [PACKFILE]\n> +\n> +----\n> + acknowledgments section\n> +\t* Always begins with the section header \"acknowledgments\"\n> +\n> +\t* The server will respond with \"NAK\" if none of the object ids sent\n> +\t as have lines were common.\n> +\n> +\t* The server will respond with \"ACK obj-id\" for all of the\n> +\t object ids sent as have lines which are common.\n> +\n> +\t* A response cannot have both \"ACK\" lines as well as a \"NAK\"\n> +\t line.\n> +\n> +\t* The server will respond with a \"ready\" line indicating that\n> +\t the server has found an acceptable common base and is ready to\n> +\t make and send a packfile (which will be found in the packfile\n> +\t section of the same response)\n> +\n> +\t* If the client determines that it is finished with negotiations\n> +\t by sending a \"done\" line, the acknowledgments sections can be\n> +\t omitted from the server's response as an optimization.\n\nShould this be changed to \"must\"? The current implementation does not\nsupport it (on the client side).\n\n> +#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }\n\nOptional: the trailing zeroes can be omitted. (That's shorter, and also\neasier to maintain when we add new fields.)\n\n> +int upload_pack_v2(struct repository *r, struct argv_array *keys,\n> +\t\t struct argv_array *args)\n> +{\n> +\tenum fetch_state state = FETCH_PROCESS_ARGS;\n> +\tstruct upload_pack_data data = UPLOAD_PACK_DATA_INIT;\n> +\tuse_sideband = LARGE_PACKET_MAX;\n> +\n> +\twhile (state != FETCH_DONE) {\n> +\t\tswitch (state) {\n> +\t\tcase FETCH_PROCESS_ARGS:\n> +\t\t\tprocess_args(args, &data);\n> +\n> +\t\t\tif (!want_obj.nr) {\n> +\t\t\t\t/*\n> +\t\t\t\t * Request didn't contain any 'want' lines,\n> +\t\t\t\t * guess they didn't want anything.\n> +\t\t\t\t */\n> +\t\t\t\tstate = FETCH_DONE;\n> +\t\t\t} else if (data.haves.nr) {\n> +\t\t\t\t/*\n> +\t\t\t\t * Request had 'have' lines, so lets ACK them.\n> +\t\t\t\t */\n> +\t\t\t\tstate = FETCH_SEND_ACKS;\n> +\t\t\t} else {\n> +\t\t\t\t/*\n> +\t\t\t\t * Request had 'want's but no 'have's so we can\n> +\t\t\t\t * immedietly go to construct and send a pack.\n> +\t\t\t\t */\n> +\t\t\t\tstate = FETCH_SEND_PACK;\n> +\t\t\t}\n> +\t\t\tbreak;\n> +\t\tcase FETCH_READ_HAVES:\n> +\t\t\tread_haves(&data);\n> +\t\t\tstate = FETCH_SEND_ACKS;\n> +\t\t\tbreak;\n\nThis branch seems to never be taken?\n\n> +\t\tcase FETCH_SEND_ACKS:\n> +\t\t\tif (process_haves_and_send_acks(&data))\n> +\t\t\t\tstate = FETCH_SEND_PACK;\n> +\t\t\telse\n> +\t\t\t\tstate = FETCH_DONE;\n> +\t\t\tbreak;\n> +\t\tcase FETCH_SEND_PACK:\n> +\t\t\tpacket_write_fmt(1, \"packfile\\n\");\n> +\t\t\tcreate_pack_file();\n> +\t\t\tstate = FETCH_DONE;\n> +\t\t\tbreak;\n> +\t\tcase FETCH_DONE:\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t}\n> +\n> +\tupload_pack_data_clear(&data);\n> +\treturn 0;\n> +}\n"},{"id":"339868","messageId":"20180221155143.02c9a1c15e54f13ae99464d4@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-25-bmwill@google.com","subject":"Re: [PATCH v3 24/35] connect: refactor git_connect to only get the protocol version once","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-21T23:51:43Z","receivedAt":"2018-02-21T23:51:51Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:13:01 -0800\nBrandon Williams wrote:\n\n> Instead of having each builtin transport asking for which protocol\n> version the user has configured in 'protocol.version' by calling\n> `get_protocol_version_config()` multiple times, factor this logic out\n> so there is just a single call at the beginning of `git_connect()`.\n> \n> This will be helpful in the next patch where we can have centralized\n> logic which determines if we need to request a different protocol\n> version than what the user has configured.\n> \n> Signed-off-by: Brandon Williams \n\nReviewed-by: Jonathan Tan \n"},{"id":"339872","messageId":"20180221160116.1619fcc4ffa7e9c13a5ab9ba@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-29-bmwill@google.com","subject":"Re: [PATCH v3 28/35] transport-helper: introduce stateless-connect","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-22T00:01:16Z","receivedAt":"2018-02-22T00:01:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:13:05 -0800\nBrandon Williams wrote:\n\n> Introduce the transport-helper capability 'stateless-connect'. This\n> capability indicates that the transport-helper can be requested to run\n> the 'stateless-connect' command which should attempt to make a\n> stateless connection with a remote end. Once established, the\n> connection can be used by the git client to communicate with\n> the remote end natively in a stateless-rpc manner as supported by\n> protocol v2. This means that the client must send everything the server\n> needs in a single request as the client must not assume any\n> state-storing on the part of the server or transport.\n\nMaybe it's worth mentioning that support in the actual remote helpers\nwill be added in a subsequent patch.\n\n> If a stateless connection cannot be established then the remote-helper\n> will respond in the same manner as the 'connect' command indicating that\n> the client should fallback to using the dumb remote-helper commands.\n\nThis makes sense, but there doesn't seem to be any code in this patch\nthat implements this.\n\n> @@ -612,6 +615,11 @@ static int process_connect_service(struct transport *transport,\n> \tif (data->connect) {\n> \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n> \t\tret = run_connect(transport, &cmdbuf);\n> +\t} else if (data->stateless_connect) {\n> +\t\tstrbuf_addf(&cmdbuf, \"stateless-connect %s\\n\", name);\n> +\t\tret = run_connect(transport, &cmdbuf);\n> +\t\tif (ret)\n> +\t\t\ttransport->stateless_rpc = 1;\n\nWhy is process_connect_service() falling back to stateless_connect if\nconnect doesn't work? I don't think this fallback would work, as a\nclient that needs \"connect\" might need its full capabilities.\n"},{"id":"339873","messageId":"20180221160606.af3db12daeed08ba68bd37f8@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-31-bmwill@google.com","subject":"Re: [PATCH v3 30/35] remote-curl: create copy of the service name","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-22T00:06:06Z","receivedAt":"2018-02-22T00:06:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:13:07 -0800\nBrandon Williams wrote:\n\n> Make a copy of the service name being requested instead of relying on\n> the buffer pointed to by the passed in 'const char *' to remain\n> unchanged.\n> \n> Signed-off-by: Brandon Williams \n\nProbably worth mentioning in the commit message:\n\n Currently, all service names are string constants, but a subsequent\n patch will introduce service names from external sources.\n\nOther than that,\n\nReviewed-by: Jonathan Tan \n"},{"id":"339874","messageId":"20180221160905.5772106219ded5f16919de46@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-33-bmwill@google.com","subject":"Re: [PATCH v3 32/35] http: allow providing extra headers for http requests","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-22T00:09:05Z","receivedAt":"2018-02-22T00:09:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:13:09 -0800\nBrandon Williams wrote:\n\n> @@ -172,6 +172,8 @@ struct http_get_options {\n> \t * for details.\n> \t */\n> \tstruct strbuf *base_url;\n> +\n> +\tstruct string_list *extra_headers;\n\nDocument this? For example:\n\n If not NULL, additional HTTP headers to be sent with the request. The\n strings in the list must not be freed until after the request.\n"},{"id":"339875","messageId":"20180221161209.80f0a3b941a069ea78ccf43e@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-36-bmwill@google.com","subject":"Re: [PATCH v3 35/35] remote-curl: don't request v2 when pushing","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-22T00:12:09Z","receivedAt":"2018-02-22T00:12:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:13:12 -0800\nBrandon Williams wrote:\n\n> +test_expect_success 'push with http:// and a config of v2 does not request v2' '\n> +\t# Till v2 for push is designed, make sure that if a client has\n> +\t# protocol.version configured to use v2, that the client instead falls\n> +\t# back and uses v0.\n> +\n> +\ttest_commit -C http_child three &&\n> +\n> +\t# Push to another branch, as the target repository has the\n> +\t# master branch checked out and we cannot push into it.\n> +\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n> +\t\tpush origin HEAD:client_branch && 2>log &&\n\nShould it be protocol.version=2? Also, two double ampersands?\n\nAlso, optionally, it might be better to do\nGIT_TRACE_PACKET=\"$(pwd)/log\", so that it does not get mixed with other\nstderr output.\n"},{"id":"339915","messageId":"20180222093327.GA12442@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180207011312.189834-13-bmwill@google.com","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T09:33:27Z","receivedAt":"2018-02-22T09:33:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 06, 2018 at 05:12:49PM -0800, Brandon Williams wrote:\n\n> +In protocol v2 communication is command oriented. When first contacting a\n> +server a list of capabilities will advertised. Some of these capabilities\n> +will be commands which a client can request be executed. Once a command\n> +has completed, a client can reuse the connection and request that other\n> +commands be executed.\n\nIf I understand this correctly, we'll potentially have a lot more\nround-trips between the client and server (one per \"command\"). And for\ngit-over-http, each one will be its own HTTP request?\n\nWe've traditionally tried to minimize HTTP requests, but I guess it's\nnot too bad if we can keep the connection open in most cases. Then we\njust suffer some extra framing bytes, but we don't have to re-establish\nthe TCP connection each time.\n\nI do wonder if the extra round trips will be noticeable in high-latency\nconditions. E.g., if I'm 200ms away, converting the current\nref-advertisement spew to \"capabilities, then the client asks for refs,\nthen we spew the refs\" is going to cost an extra 200ms, even if the\nfetch just ends up being a noop. I'm not sure how bad that is in the\ngrand scheme of things (after all, the TCP handshake involves some\nround-trips, too).\n\n> + Capability Advertisement\n> +--------------------------\n> +\n> +A server which decides to communicate (based on a request from a client)\n> +using protocol version 2, notifies the client by sending a version string\n> +in its initial response followed by an advertisement of its capabilities.\n> +Each capability is a key with an optional value. Clients must ignore all\n> +unknown keys. Semantics of unknown values are left to the definition of\n> +each key. Some capabilities will describe commands which can be requested\n> +to be executed by the client.\n> +\n> + capability-advertisement = protocol-version\n> +\t\t\t capability-list\n> +\t\t\t flush-pkt\n> +\n> + protocol-version = PKT-LINE(\"version 2\" LF)\n> + capability-list = *capability\n> + capability = PKT-LINE(key[=value] LF)\n> +\n> + key = 1*CHAR\n> + value = 1*CHAR\n> + CHAR = 1*(ALPHA / DIGIT / \"-\" / \"_\")\n> +\n> +A client then responds to select the command it wants with any particular\n> +capabilities or arguments. There is then an optional section where the\n> +client can provide any command specific parameters or queries.\n> +\n> + command-request = command\n> +\t\t capability-list\n> +\t\t (command-args)\n> +\t\t flush-pkt\n> + command = PKT-LINE(\"command=\" key LF)\n> + command-args = delim-pkt\n> +\t\t *arg\n> + arg = 1*CHAR\n\nFor a single stateful TCP connection like git:// or git-over-ssh, the\nclient would get the capabilities once and then issue a series of\ncommands. For git-over-http, how does it work?\n\nThe client speaks first in HTTP, so we'd first make a request to get\njust the capabilities from the server? And then proceed from there with\na series of requests, assuming that the capabilities for each server we\nsubsequently contact are the same? That's probably reasonable (and\ncertainly the existing http protocol makes that capabilities\nassumption).\n\nI don't see any documentation on how this all works with http. But\nreading patch 34, it looks like we just do the usual\nservice=git-upload-pack request (with the magic request for v2), and\nthen the server would send us capabilities. Which follows my line of\nthinking in the paragraph above.\n\n-Peff\n"},{"id":"339916","messageId":"20180222094831.GB12442@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180207011312.189834-14-bmwill@google.com","subject":"Re: [PATCH v3 13/35] ls-refs: introduce ls-refs server command","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T09:48:31Z","receivedAt":"2018-02-22T09:48:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 06, 2018 at 05:12:50PM -0800, Brandon Williams wrote:\n\n> +ls-refs takes in the following parameters wrapped in packet-lines:\n> +\n> + symrefs\n> +\tIn addition to the object pointed by it, show the underlying ref\n> +\tpointed by it when showing a symbolic ref.\n> + peel\n> +\tShow peeled tags.\n> + ref-pattern \n> +\tWhen specified, only references matching the one of the provided\n> +\tpatterns are displayed.\n\nHow do we match those patterns? That's probably an important thing to\ninclude in the spec.\n\nLooking at the code, I see:\n\n> +/*\n> + * Check if one of the patterns matches the tail part of the ref.\n> + * If no patterns were provided, all refs match.\n> + */\n> +static int ref_match(const struct argv_array *patterns, const char *refname)\n\nThis kind of tail matching can't quite implement all of the current\nbehavior. Because we actually do the normal dwim_ref() matching, which\nincludes stuff like \"refs/remotes/%s/HEAD\".\n\nThe other problem with tail-matching is that it's inefficient on the\nserver. Ideally we could get a request for \"master\" and only look up\nrefs/heads/master, refs/tags/master, etc. And if there are 50,000 refs\nin refs/pull, we wouldn't have to process those at all. Of course this\nis no worse than the current code, which not only looks at each ref but\nactually _sends_ it. But it would be nice if we could fix this.\n\nThere's some more discussion in this old thread:\n\n https://public-inbox.org/git/20161024132932.i42rqn2vlpocqmkq@sigill.intra.peff.net/\n\n> +{\n> +\tchar *pathbuf;\n> +\tint i;\n> +\n> +\tif (!patterns->argc)\n> +\t\treturn 1; /* no restriction */\n> +\n> +\tpathbuf = xstrfmt(\"/%s\", refname);\n> +\tfor (i = 0; i < patterns->argc; i++) {\n> +\t\tif (!wildmatch(patterns->argv[i], pathbuf, 0)) {\n> +\t\t\tfree(pathbuf);\n> +\t\t\treturn 1;\n> +\t\t}\n> +\t}\n> +\tfree(pathbuf);\n> +\treturn 0;\n> +}\n\nDoes the client have to be aware that we're using wildmatch? I think\nthey'd need \"refs/heads/**\" to actually implement what we usually\nspecify in refspecs as \"refs/heads/*\". Or does the lack of WM_PATHNAME\nmake this work with just \"*\"?\n\nDo we anticipate that the client would left-anchor the refspec like\n\"/refs/heads/*\" so that in theory the server could avoid looking outside\nof /refs/heads/?\n\n-Peff\n"},{"id":"339919","messageId":"20180222095833.GC12442@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180221134422.2386e1aca39fe673235590e7@google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T09:58:34Z","receivedAt":"2018-02-22T09:58:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 21, 2018 at 01:44:22PM -0800, Jonathan Tan wrote:\n\n> On Tue, 6 Feb 2018 17:12:41 -0800\n> Brandon Williams wrote:\n> \n> > In order to allow for code sharing with the server-side of fetch in\n> > protocol-v2 convert upload-pack to be a builtin.\n> > \n> > Signed-off-by: Brandon Williams \n> \n> As Stefan mentioned in [1], also mention in the commit message that this\n> means that the \"git-upload-pack\" invocation gains additional\n> capabilities (for example, invoking a pager for --help).\n\nAnd possibly respecting pager.upload-pack, which would violate our rule\nthat it is safe to run upload-pack in untrusted repositories.\n\n(This actually doesn't work right now because pager.* is broken for\nbuiltins that don't specify RUN_SETUP; but I think with the fixes last\nyear to the config code, we can now drop that restriction).\n\nObviously we can work around this with an extra RUN_NO_PAGER_CONFIG\nflag. But I think it points to a general danger in making upload-pack a\nbuiltin. I'm not sure what other features it would want to avoid (or\nwhat might grow in the future).\n\n> Having said that, the main purpose of this patch seems to be to libify\n> upload-pack, and the move to builtin is just a way of putting the\n> program somewhere - we could have easily renamed upload-pack.c and\n> created a new upload-pack.c containing the main(), preserving the\n> non-builtin-ness of upload-pack, while still gaining the benefits of\n> libifying upload-pack.\n\nYeah, this seems like a better route to me.\n\n-Peff\n"},{"id":"339927","messageId":"20180222180715.GB185096@google.com","threadId":"47524","inReplyTo":"20180222095833.GC12442@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:07:15Z","receivedAt":"2018-02-22T18:07:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Jeff King wrote:\n> On Wed, Feb 21, 2018 at 01:44:22PM -0800, Jonathan Tan wrote:\n> \n> > On Tue, 6 Feb 2018 17:12:41 -0800\n> > Brandon Williams wrote:\n> > \n> > > In order to allow for code sharing with the server-side of fetch in\n> > > protocol-v2 convert upload-pack to be a builtin.\n> > > \n> > > Signed-off-by: Brandon Williams \n> > \n> > As Stefan mentioned in [1], also mention in the commit message that this\n> > means that the \"git-upload-pack\" invocation gains additional\n> > capabilities (for example, invoking a pager for --help).\n> \n> And possibly respecting pager.upload-pack, which would violate our rule\n> that it is safe to run upload-pack in untrusted repositories.\n\nAnd this isn't an issue with receive-pack because this same guarantee\ndoesn't exist?\n\n> \n> (This actually doesn't work right now because pager.* is broken for\n> builtins that don't specify RUN_SETUP; but I think with the fixes last\n> year to the config code, we can now drop that restriction).\n> \n> Obviously we can work around this with an extra RUN_NO_PAGER_CONFIG\n> flag. But I think it points to a general danger in making upload-pack a\n> builtin. I'm not sure what other features it would want to avoid (or\n> what might grow in the future).\n> \n> > Having said that, the main purpose of this patch seems to be to libify\n> > upload-pack, and the move to builtin is just a way of putting the\n> > program somewhere - we could have easily renamed upload-pack.c and\n> > created a new upload-pack.c containing the main(), preserving the\n> > non-builtin-ness of upload-pack, while still gaining the benefits of\n> > libifying upload-pack.\n> \n> Yeah, this seems like a better route to me.\n> \n> -Peff\n\n-- \nBrandon Williams\n"},{"id":"339928","messageId":"20180222181400.GA19035@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180222180715.GB185096@google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T18:14:00Z","receivedAt":"2018-02-22T18:14:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 10:07:15AM -0800, Brandon Williams wrote:\n\n> On 02/22, Jeff King wrote:\n> > On Wed, Feb 21, 2018 at 01:44:22PM -0800, Jonathan Tan wrote:\n> > \n> > > On Tue, 6 Feb 2018 17:12:41 -0800\n> > > Brandon Williams wrote:\n> > > \n> > > > In order to allow for code sharing with the server-side of fetch in\n> > > > protocol-v2 convert upload-pack to be a builtin.\n> > > > \n> > > > Signed-off-by: Brandon Williams \n> > > \n> > > As Stefan mentioned in [1], also mention in the commit message that this\n> > > means that the \"git-upload-pack\" invocation gains additional\n> > > capabilities (for example, invoking a pager for --help).\n> > \n> > And possibly respecting pager.upload-pack, which would violate our rule\n> > that it is safe to run upload-pack in untrusted repositories.\n> \n> And this isn't an issue with receive-pack because this same guarantee\n> doesn't exist?\n\nYes, exactly (which is confusing and weird, yes, but that's how it is).\n\n-Peff\n"},{"id":"339929","messageId":"20180222181739.GC185096@google.com","threadId":"47524","inReplyTo":"20180221141135.c78d72b422f985aae5375bca@google.com","subject":"Re: [PATCH v3 08/35] connect: discover protocol version outside of get_remote_heads","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:17:39Z","receivedAt":"2018-02-22T18:17:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:12:45 -0800\n> Brandon Williams wrote:\n> \n> > -\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n> > +\n> > +\tpacket_reader_init(&reader, fd[0], NULL, 0,\n> > +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> > +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > +\n> > +\tswitch (discover_version(&reader)) {\n> > +\tcase protocol_v1:\n> > +\tcase protocol_v0:\n> > +\t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n> > +\t\tbreak;\n> > +\tcase protocol_unknown_version:\n> > +\t\tBUG(\"unknown protocol version\");\n> > +\t}\n> \n> This inlining is repeated a few times, which raises the question: if the\n> intention is to keep the v0/1 logic separately from v2, why not have a\n> single function that wraps them all? Looking at the end result (after\n> all the patches in this patch set are applied), it seems that the v2\n> version does not have extra_have or shallow parameters, which is a good\n> enough reason for me (I don't think functions that take in many\n> arguments and then selectively use them is a good idea). I think that\n> other reviewers will have this question too, so maybe discuss this in\n> the commit message.\n\nYes this sort of switch statement appears a few times but really there\nisn't a good way to \"have one function to wrap it all\" with the current\nstate of the code. That sort of change would take tons of refactoring to\nget into a state where we could do that, and is outside the scope of\nthis series.\n\n> \n> > diff --git a/remote.h b/remote.h\n> > index 1f6611be2..2016461df 100644\n> > --- a/remote.h\n> > +++ b/remote.h\n> > @@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n> > void free_refs(struct ref *ref);\n> > \n> > struct oid_array;\n> > -extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > +struct packet_reader;\n> > +extern struct ref **get_remote_heads(struct packet_reader *reader,\n> > \t\t\t\t struct ref **list, unsigned int flags,\n> > \t\t\t\t struct oid_array *extra_have,\n> > -\t\t\t\t struct oid_array *shallow);\n> > +\t\t\t\t struct oid_array *shallow_points);\n> \n> This change probably does not belong in this patch, especially since\n> remote.c is unchanged.\n\nYes this hunk is needed, the signature of get_remote_heads changes. It\nmay be difficult to see that due to the fact that we don't really have a\nclear story on how header files are divided up within the project.\n\n-- \nBrandon Williams\n"},{"id":"339931","messageId":"20180222181922.GD185096@google.com","threadId":"47524","inReplyTo":"20180221145411.35b2ea84747518a499276bdd@google.com","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:19:22Z","receivedAt":"2018-02-22T18:19:32Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:12:51 -0800\n> Brandon Williams wrote:\n> \n> > +extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> > +\t\t\t\t struct ref **list, int for_push,\n> > +\t\t\t\t const struct argv_array *ref_patterns);\n> \n> I haven't looked at the rest of this patch in detail, but the type of\n> ref_patterns is probably better as struct string_list, since this is not\n> a true argument array (e.g. with flags starting with --). Same comment\n> for the next few patches that deal with ref patterns.\n\nIts just a list of strings which don't require having a util pointer\nhanging around so actually using an argv_array would be more memory\nefficient than a string_list. But either way I don't think it matters\nmuch.\n\n-- \nBrandon Williams\n"},{"id":"339934","messageId":"20180222182538.GE185096@google.com","threadId":"47524","inReplyTo":"20180221145639.c6cf2409ce2120109bdd169f@google.com","subject":"Re: [PATCH v3 15/35] transport: convert get_refs_list to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:25:38Z","receivedAt":"2018-02-22T18:25:45Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:12:52 -0800\n> Brandon Williams wrote:\n> \n> > @@ -21,7 +22,8 @@ struct transport_vtable {\n> > \t * the ref without a huge amount of effort, it should store it\n> > \t * in the ref's old_sha1 field; otherwise it should be all 0.\n> > \t **/\n> > -\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push);\n> > +\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n> > +\t\t\t\t const struct argv_array *ref_patterns);\n> \n> Also mention in the documentation that this function is allowed to\n> return refs that do not match the ref patterns.\n\nI'll add a comment.\n\n-- \nBrandon Williams\n"},{"id":"339935","messageId":"20180222182647.GF185096@google.com","threadId":"47524","inReplyTo":"20180221145804.f483fb21559f71618812404d@google.com","subject":"Re: [PATCH v3 16/35] transport: convert transport_get_remote_refs to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:26:47Z","receivedAt":"2018-02-22T18:26:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:12:53 -0800\n> Brandon Williams wrote:\n> \n> > -const struct ref *transport_get_remote_refs(struct transport *transport)\n> > +const struct ref *transport_get_remote_refs(struct transport *transport,\n> > +\t\t\t\t\t const struct argv_array *ref_patterns)\n> > {\n> > \tif (!transport->got_remote_refs) {\n> > -\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n> > +\t\ttransport->remote_refs =\n> > +\t\t\ttransport->vtable->get_refs_list(transport, 0,\n> > +\t\t\t\t\t\t\t ref_patterns);\n> > \t\ttransport->got_remote_refs = 1;\n> > \t}\n> \n> Should we do our own client-side filtering if the server side cannot do\n> it for us (because it doesn't support protocol v2)? Either way, this\n> decision should be mentioned in the commit message.\n\nIf someone wants to add this in the future they can, but that is outside\nthe scope of this series.\n\n-- \nBrandon Williams\n"},{"id":"339936","messageId":"20180222182657.GE19035@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180222181922.GD185096@google.com","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T18:26:58Z","receivedAt":"2018-02-22T18:27:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 10:19:22AM -0800, Brandon Williams wrote:\n\n> On 02/21, Jonathan Tan wrote:\n> > On Tue, 6 Feb 2018 17:12:51 -0800\n> > Brandon Williams wrote:\n> > \n> > > +extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> > > +\t\t\t\t struct ref **list, int for_push,\n> > > +\t\t\t\t const struct argv_array *ref_patterns);\n> > \n> > I haven't looked at the rest of this patch in detail, but the type of\n> > ref_patterns is probably better as struct string_list, since this is not\n> > a true argument array (e.g. with flags starting with --). Same comment\n> > for the next few patches that deal with ref patterns.\n> \n> Its just a list of strings which don't require having a util pointer\n> hanging around so actually using an argv_array would be more memory\n> efficient than a string_list. But either way I don't think it matters\n> much.\n\nI agree that it shouldn't matter much here. But if the name argv_array\nis standing in the way of using it, I think we should consider giving it\na more general name. I picked that not to evoke \"this must be arguments\"\nbut \"this is terminated by a single NULL\".\n\nIn general I think it should be the preferred structure for string\nlists, just because it actually converts for free to the \"other\" common\nformat (whereas you can never pass string_list.items to a function that\ndoesn't know about string lists).\n\n-Peff\n"},{"id":"339942","messageId":"20180222185353.GH185096@google.com","threadId":"47524","inReplyTo":"20180221160116.1619fcc4ffa7e9c13a5ab9ba@google.com","subject":"Re: [PATCH v3 28/35] transport-helper: introduce stateless-connect","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:53:53Z","receivedAt":"2018-02-22T18:54:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:13:05 -0800\n> Brandon Williams wrote:\n> \n> > Introduce the transport-helper capability 'stateless-connect'. This\n> > capability indicates that the transport-helper can be requested to run\n> > the 'stateless-connect' command which should attempt to make a\n> > stateless connection with a remote end. Once established, the\n> > connection can be used by the git client to communicate with\n> > the remote end natively in a stateless-rpc manner as supported by\n> > protocol v2. This means that the client must send everything the server\n> > needs in a single request as the client must not assume any\n> > state-storing on the part of the server or transport.\n> \n> Maybe it's worth mentioning that support in the actual remote helpers\n> will be added in a subsequent patch.\n\nI can mention that.\n\n> \n> > If a stateless connection cannot be established then the remote-helper\n> > will respond in the same manner as the 'connect' command indicating that\n> > the client should fallback to using the dumb remote-helper commands.\n> \n> This makes sense, but there doesn't seem to be any code in this patch\n> that implements this.\n> \n> > @@ -612,6 +615,11 @@ static int process_connect_service(struct transport *transport,\n> > \tif (data->connect) {\n> > \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n> > \t\tret = run_connect(transport, &cmdbuf);\n> > +\t} else if (data->stateless_connect) {\n> > +\t\tstrbuf_addf(&cmdbuf, \"stateless-connect %s\\n\", name);\n> > +\t\tret = run_connect(transport, &cmdbuf);\n> > +\t\tif (ret)\n> > +\t\t\ttransport->stateless_rpc = 1;\n> \n> Why is process_connect_service() falling back to stateless_connect if\n> connect doesn't work? I don't think this fallback would work, as a\n> client that needs \"connect\" might need its full capabilities.\n\nRight now there isn't really a notion of \"needing\" connect since if\nconnect fails then you need to fallback to doing the dumb thing. Also\nnote that there isn't all fallback from connect to stateless-connect\nhere. If the remote helper advertises connect, only connect will be\ntried even if stateless-connect is advertised. So this only really\nworks in the case where stateless-connect is advertised and connect\nisn't, as is with our http remote-helper.\n\n-- \nBrandon Williams\n"},{"id":"339943","messageId":"20180222184837.GG185096@google.com","threadId":"47524","inReplyTo":"20180221154622.f35e69f0d8c6bb54e40a299a@google.com","subject":"Re: [PATCH v3 20/35] upload-pack: introduce fetch server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:48:37Z","receivedAt":"2018-02-22T18:54:31Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:12:57 -0800\n> Brandon Williams wrote:\n> \n> > + want \n> > +\tIndicates to the server an object which the client wants to\n> > +\tretrieve.\n> \n> Mention that the client can \"want\" anything even if not advertised by\n> the server (like uploadpack.allowanysha1inwant).\n\nWill do.\n\n> \n> > + output = *section\n> > + section = (acknowledgments | packfile)\n> > +\t (flush-pkt | delim-pkt)\n> > +\n> > + acknowledgments = PKT-LINE(\"acknowledgments\" LF)\n> > +\t\t *(ready | nak | ack)\n> \n> Can this part be described more precisely in the BNF section? I see that\n> you describe later that there can be multiple ACKs or one NAK (but not\n> both), and \"ready\" can be sent regardless of whether ACKs or a NAK is\n> sent.\n\nYep I'll fix that.\n\n> \n> > + ready = PKT-LINE(\"ready\" LF)\n> > + nak = PKT-LINE(\"NAK\" LF)\n> > + ack = PKT-LINE(\"ACK\" SP obj-id LF)\n> > +\n> > + packfile = PKT-LINE(\"packfile\" LF)\n> > +\t [PACKFILE]\n> > +\n> > +----\n> > + acknowledgments section\n> > +\t* Always begins with the section header \"acknowledgments\"\n> > +\n> > +\t* The server will respond with \"NAK\" if none of the object ids sent\n> > +\t as have lines were common.\n> > +\n> > +\t* The server will respond with \"ACK obj-id\" for all of the\n> > +\t object ids sent as have lines which are common.\n> > +\n> > +\t* A response cannot have both \"ACK\" lines as well as a \"NAK\"\n> > +\t line.\n> > +\n> > +\t* The server will respond with a \"ready\" line indicating that\n> > +\t the server has found an acceptable common base and is ready to\n> > +\t make and send a packfile (which will be found in the packfile\n> > +\t section of the same response)\n> > +\n> > +\t* If the client determines that it is finished with negotiations\n> > +\t by sending a \"done\" line, the acknowledgments sections can be\n> > +\t omitted from the server's response as an optimization.\n> \n> Should this be changed to \"must\"? The current implementation does not\n> support it (on the client side).\n\nThis is actually a great question and one which may need to be thought\nabout in terms of its application to future extensions to the fetch\ncommand. Since fetch's response is now broken up into sections we may\nwant the client to cope with sections being in any order and maybe even\nskipping sections it doesn't know about. Not sure if its necessary but\nits an idea.\n\n> \n> > +#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }\n> \n> Optional: the trailing zeroes can be omitted. (That's shorter, and also\n> easier to maintain when we add new fields.)\n> \n> > +int upload_pack_v2(struct repository *r, struct argv_array *keys,\n> > +\t\t struct argv_array *args)\n> > +{\n> > +\tenum fetch_state state = FETCH_PROCESS_ARGS;\n> > +\tstruct upload_pack_data data = UPLOAD_PACK_DATA_INIT;\n> > +\tuse_sideband = LARGE_PACKET_MAX;\n> > +\n> > +\twhile (state != FETCH_DONE) {\n> > +\t\tswitch (state) {\n> > +\t\tcase FETCH_PROCESS_ARGS:\n> > +\t\t\tprocess_args(args, &data);\n> > +\n> > +\t\t\tif (!want_obj.nr) {\n> > +\t\t\t\t/*\n> > +\t\t\t\t * Request didn't contain any 'want' lines,\n> > +\t\t\t\t * guess they didn't want anything.\n> > +\t\t\t\t */\n> > +\t\t\t\tstate = FETCH_DONE;\n> > +\t\t\t} else if (data.haves.nr) {\n> > +\t\t\t\t/*\n> > +\t\t\t\t * Request had 'have' lines, so lets ACK them.\n> > +\t\t\t\t */\n> > +\t\t\t\tstate = FETCH_SEND_ACKS;\n> > +\t\t\t} else {\n> > +\t\t\t\t/*\n> > +\t\t\t\t * Request had 'want's but no 'have's so we can\n> > +\t\t\t\t * immedietly go to construct and send a pack.\n> > +\t\t\t\t */\n> > +\t\t\t\tstate = FETCH_SEND_PACK;\n> > +\t\t\t}\n> > +\t\t\tbreak;\n> > +\t\tcase FETCH_READ_HAVES:\n> > +\t\t\tread_haves(&data);\n> > +\t\t\tstate = FETCH_SEND_ACKS;\n> > +\t\t\tbreak;\n> \n> This branch seems to never be taken?\n\nMust be left over from another version, I'll remove it.\n\n> \n> > +\t\tcase FETCH_SEND_ACKS:\n> > +\t\t\tif (process_haves_and_send_acks(&data))\n> > +\t\t\t\tstate = FETCH_SEND_PACK;\n> > +\t\t\telse\n> > +\t\t\t\tstate = FETCH_DONE;\n> > +\t\t\tbreak;\n> > +\t\tcase FETCH_SEND_PACK:\n> > +\t\t\tpacket_write_fmt(1, \"packfile\\n\");\n> > +\t\t\tcreate_pack_file();\n> > +\t\t\tstate = FETCH_DONE;\n> > +\t\t\tbreak;\n> > +\t\tcase FETCH_DONE:\n> > +\t\t\tcontinue;\n> > +\t\t}\n> > +\t}\n> > +\n> > +\tupload_pack_data_clear(&data);\n> > +\treturn 0;\n> > +}\n\n-- \nBrandon Williams\n"},{"id":"339944","messageId":"20180222185652.GI185096@google.com","threadId":"47524","inReplyTo":"20180221160606.af3db12daeed08ba68bd37f8@google.com","subject":"Re: [PATCH v3 30/35] remote-curl: create copy of the service name","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:56:52Z","receivedAt":"2018-02-22T18:57:00Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:13:07 -0800\n> Brandon Williams wrote:\n> \n> > Make a copy of the service name being requested instead of relying on\n> > the buffer pointed to by the passed in 'const char *' to remain\n> > unchanged.\n> > \n> > Signed-off-by: Brandon Williams \n> \n> Probably worth mentioning in the commit message:\n> \n> Currently, all service names are string constants, but a subsequent\n> patch will introduce service names from external sources.\n> \n> Other than that,\n> \n> Reviewed-by: Jonathan Tan \n\nI'll add that.\n\n-- \nBrandon Williams\n"},{"id":"339945","messageId":"20180222185853.GJ185096@google.com","threadId":"47524","inReplyTo":"20180221160905.5772106219ded5f16919de46@google.com","subject":"Re: [PATCH v3 32/35] http: allow providing extra headers for http requests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:58:53Z","receivedAt":"2018-02-22T18:59:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:13:09 -0800\n> Brandon Williams wrote:\n> \n> > @@ -172,6 +172,8 @@ struct http_get_options {\n> > \t * for details.\n> > \t */\n> > \tstruct strbuf *base_url;\n> > +\n> > +\tstruct string_list *extra_headers;\n> \n> Document this? For example:\n> \n> If not NULL, additional HTTP headers to be sent with the request. The\n> strings in the list must not be freed until after the request.\n\nI'll add that.\n\n-- \nBrandon Williams\n"},{"id":"339946","messageId":"20180222185948.GK185096@google.com","threadId":"47524","inReplyTo":"20180221161209.80f0a3b941a069ea78ccf43e@google.com","subject":"Re: [PATCH v3 35/35] remote-curl: don't request v2 when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T18:59:48Z","receivedAt":"2018-02-22T18:59:55Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:13:12 -0800\n> Brandon Williams wrote:\n> \n> > +test_expect_success 'push with http:// and a config of v2 does not request v2' '\n> > +\t# Till v2 for push is designed, make sure that if a client has\n> > +\t# protocol.version configured to use v2, that the client instead falls\n> > +\t# back and uses v0.\n> > +\n> > +\ttest_commit -C http_child three &&\n> > +\n> > +\t# Push to another branch, as the target repository has the\n> > +\t# master branch checked out and we cannot push into it.\n> > +\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n> > +\t\tpush origin HEAD:client_branch && 2>log &&\n> \n> Should it be protocol.version=2? Also, two double ampersands?\n> \n> Also, optionally, it might be better to do\n> GIT_TRACE_PACKET=\"$(pwd)/log\", so that it does not get mixed with other\n> stderr output.\n\nWow thanks for catching that, let me fix that.\n\n-- \nBrandon Williams\n"},{"id":"339947","messageId":"20180222190901.GL185096@google.com","threadId":"47524","inReplyTo":"20180222185948.GK185096@google.com","subject":"Re: [PATCH v3 35/35] remote-curl: don't request v2 when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T19:09:01Z","receivedAt":"2018-02-22T19:09:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Brandon Williams wrote:\n> On 02/21, Jonathan Tan wrote:\n> > On Tue, 6 Feb 2018 17:13:12 -0800\n> > Brandon Williams wrote:\n> > \n> > > +test_expect_success 'push with http:// and a config of v2 does not request v2' '\n> > > +\t# Till v2 for push is designed, make sure that if a client has\n> > > +\t# protocol.version configured to use v2, that the client instead falls\n> > > +\t# back and uses v0.\n> > > +\n> > > +\ttest_commit -C http_child three &&\n> > > +\n> > > +\t# Push to another branch, as the target repository has the\n> > > +\t# master branch checked out and we cannot push into it.\n> > > +\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n> > > +\t\tpush origin HEAD:client_branch && 2>log &&\n> > \n> > Should it be protocol.version=2? Also, two double ampersands?\n> > \n> > Also, optionally, it might be better to do\n> > GIT_TRACE_PACKET=\"$(pwd)/log\", so that it does not get mixed with other\n> > stderr output.\n> \n> Wow thanks for catching that, let me fix that.\n\nI like setting the log via \"$(pwd)/log\" but it turns out that this\nappends to the file if it already exists, which means the previous tests\nneed to do some cleanup. This is actually probably preferable anyway.\n\n> \n> -- \n> Brandon Williams\n\n-- \nBrandon Williams\n"},{"id":"339948","messageId":"CAGZ79kauYniFpo+Y_hFauVKMWNgxjQ_OOoX0mgePcr3Z6hSLWg@mail.gmail.com","threadId":"47524","inReplyTo":"20180207011312.189834-4-bmwill@google.com","subject":"Re: [PATCH v3 03/35] pkt-line: add delim packet support","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-22T19:13:34Z","receivedAt":"2018-02-22T19:13:42Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> One of the design goals of protocol-v2 is to improve the semantics of\n> flush packets. Currently in protocol-v1, flush packets are used both to\n> indicate a break in a list of packet lines as well as an indication that\n> one side has finished speaking. This makes it particularly difficult\n> to implement proxies as a proxy would need to completely understand git\n> protocol instead of simply looking for a flush packet.\n>\n> To do this, introduce the special deliminator packet '0001'. A delim\n> packet can then be used as a deliminator between lists of packet lines\n> while flush packets can be reserved to indicate the end of a response.\n\nPlease mention where this can be found in the documentation.\n(Defer to later patch?)\nAs the commit message states, this is only to be used for v2,\nin v0 it is still an illegal pkt.\n\n>\n> Signed-off-by: Brandon Williams \n\nThe code is\nReviewed-by: Stefan Beller \n"},{"id":"339949","messageId":"20180222112258.7a71601780be9e42f54c5f61@google.com","threadId":"47524","inReplyTo":"20180222181739.GC185096@google.com","subject":"Re: [PATCH v3 08/35] connect: discover protocol version outside of get_remote_heads","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-22T19:22:58Z","receivedAt":"2018-02-22T19:23:07Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 22 Feb 2018 10:17:39 -0800\nBrandon Williams wrote:\n\n> > > diff --git a/remote.h b/remote.h\n> > > index 1f6611be2..2016461df 100644\n> > > --- a/remote.h\n> > > +++ b/remote.h\n> > > @@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n> > > void free_refs(struct ref *ref);\n> > > \n> > > struct oid_array;\n> > > -extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > > +struct packet_reader;\n> > > +extern struct ref **get_remote_heads(struct packet_reader *reader,\n> > > \t\t\t\t struct ref **list, unsigned int flags,\n> > > \t\t\t\t struct oid_array *extra_have,\n> > > -\t\t\t\t struct oid_array *shallow);\n> > > +\t\t\t\t struct oid_array *shallow_points);\n> > \n> > This change probably does not belong in this patch, especially since\n> > remote.c is unchanged.\n> \n> Yes this hunk is needed, the signature of get_remote_heads changes. It\n> may be difficult to see that due to the fact that we don't really have a\n> clear story on how header files are divided up within the project.\n\nThanks - I indeed didn't notice that the implementation of\nget_remote_heads() is modified too in this patch.\n\nMy initial comment was about just the renaming of \"shallow\" to\n\"shallow_points\", but yes, you're right - I see in the implementation\nthat it is indeed named \"shallow_points\" there, so this change is\njustified, especially since you're already changing the signature of\nthis function.\n"},{"id":"339950","messageId":"20180222112533.0d7c6023fb8e4098efedfe31@google.com","threadId":"47524","inReplyTo":"20180222182657.GE19035@sigill.intra.peff.net","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-22T19:25:33Z","receivedAt":"2018-02-22T19:25:41Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 22 Feb 2018 13:26:58 -0500\nJeff King wrote:\n\n> On Thu, Feb 22, 2018 at 10:19:22AM -0800, Brandon Williams wrote:\n> \n> > On 02/21, Jonathan Tan wrote:\n> > > On Tue, 6 Feb 2018 17:12:51 -0800\n> > > Brandon Williams wrote:\n> > > \n> > > > +extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> > > > +\t\t\t\t struct ref **list, int for_push,\n> > > > +\t\t\t\t const struct argv_array *ref_patterns);\n> > > \n> > > I haven't looked at the rest of this patch in detail, but the type of\n> > > ref_patterns is probably better as struct string_list, since this is not\n> > > a true argument array (e.g. with flags starting with --). Same comment\n> > > for the next few patches that deal with ref patterns.\n> > \n> > Its just a list of strings which don't require having a util pointer\n> > hanging around so actually using an argv_array would be more memory\n> > efficient than a string_list. But either way I don't think it matters\n> > much.\n> \n> I agree that it shouldn't matter much here. But if the name argv_array\n> is standing in the way of using it, I think we should consider giving it\n> a more general name. I picked that not to evoke \"this must be arguments\"\n> but \"this is terminated by a single NULL\".\n> \n> In general I think it should be the preferred structure for string\n> lists, just because it actually converts for free to the \"other\" common\n> format (whereas you can never pass string_list.items to a function that\n> doesn't know about string lists).\n\nThis sounds reasonable - I withdraw my comment about using struct\nstring_list.\n"},{"id":"339953","messageId":"CAGZ79kanWXGa3v4oJmoerh3XzisS+3V2oyO7WXK-tPTiHgSgxg@mail.gmail.com","threadId":"47524","inReplyTo":"20180207011312.189834-6-bmwill@google.com","subject":"Re: [PATCH v3 05/35] upload-pack: factor out processing lines","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-22T19:31:03Z","receivedAt":"2018-02-22T19:31:12Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> Factor out the logic for processing shallow, deepen, deepen_since, and\n> deepen_not lines into their own functions to simplify the\n> 'receive_needs()' function in addition to making it easier to reuse some\n> of this logic when implementing protocol_v2.\n>\n> Signed-off-by: Brandon Williams \n\nReviewed-by: Stefan Beller \nfor the stated purpose of just refactoring existing code for better reuse later.\n\nI do have a few comments on the code in general,\nwhich might be out of scope for this series.\n\nA close review would have been fastest if we had some sort of\nhttps://public-inbox.org/git/20171025224620.27657-1-sbeller@google.com/\nwhich I might revive soon for this purpose. (it showed that I would need it)\n\n\n> + *depth = (int)strtol(arg, &end, 0);\n\nstrtol is not used quite correctly here IMHO, as we do not\ninspect errno for ERANGE\n\nThanks,\nStefan\n"},{"id":"339954","messageId":"20180222113226.208f86fff80fb0d05c620433@google.com","threadId":"47524","inReplyTo":"20180222182647.GF185096@google.com","subject":"Re: [PATCH v3 16/35] transport: convert transport_get_remote_refs to take a list of ref patterns","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-22T19:32:26Z","receivedAt":"2018-02-22T19:32:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 22 Feb 2018 10:26:47 -0800\nBrandon Williams wrote:\n\n> On 02/21, Jonathan Tan wrote:\n> > On Tue, 6 Feb 2018 17:12:53 -0800\n> > Brandon Williams wrote:\n> > \n> > > -const struct ref *transport_get_remote_refs(struct transport *transport)\n> > > +const struct ref *transport_get_remote_refs(struct transport *transport,\n> > > +\t\t\t\t\t const struct argv_array *ref_patterns)\n> > > {\n> > > \tif (!transport->got_remote_refs) {\n> > > -\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n> > > +\t\ttransport->remote_refs =\n> > > +\t\t\ttransport->vtable->get_refs_list(transport, 0,\n> > > +\t\t\t\t\t\t\t ref_patterns);\n> > > \t\ttransport->got_remote_refs = 1;\n> > > \t}\n> > \n> > Should we do our own client-side filtering if the server side cannot do\n> > it for us (because it doesn't support protocol v2)? Either way, this\n> > decision should be mentioned in the commit message.\n> \n> If someone wants to add this in the future they can, but that is outside\n> the scope of this series.\n\nIn that case, also document that this function is allowed to return refs\nthat do not match the ref patterns.\n\nUnlike in patch 15 (which deals with the interface between the transport\ncode and transport vtables, which can be changed as long as the\ntransport code is aware of it, as I wrote in [1]), this may result in\nuser-visible differences depending on which protocol is used. But after\nmore thinking, I don't think we're in a situation yet where having extra\nrefs shown/written are harmful, and if it comes to that, we can tighten\nthis code later without backwards incompatibility. So, OK, this is fine.\n\n[1] https://public-inbox.org/git/20180221145639.c6cf2409ce2120109bdd169f@google.com/\n"},{"id":"339955","messageId":"20180222193708.GM185096@google.com","threadId":"47524","inReplyTo":"CAGZ79kauYniFpo+Y_hFauVKMWNgxjQ_OOoX0mgePcr3Z6hSLWg@mail.gmail.com","subject":"Re: [PATCH v3 03/35] pkt-line: add delim packet support","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T19:37:08Z","receivedAt":"2018-02-22T19:37:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Stefan Beller wrote:\n> On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> > One of the design goals of protocol-v2 is to improve the semantics of\n> > flush packets. Currently in protocol-v1, flush packets are used both to\n> > indicate a break in a list of packet lines as well as an indication that\n> > one side has finished speaking. This makes it particularly difficult\n> > to implement proxies as a proxy would need to completely understand git\n> > protocol instead of simply looking for a flush packet.\n> >\n> > To do this, introduce the special deliminator packet '0001'. A delim\n> > packet can then be used as a deliminator between lists of packet lines\n> > while flush packets can be reserved to indicate the end of a response.\n> \n> Please mention where this can be found in the documentation.\n> (Defer to later patch?)\n\nYeah the documentation does get added in a future patch, I'll make a\ncomment to that effect.\n\n> As the commit message states, this is only to be used for v2,\n> in v0 it is still an illegal pkt.\n> \n> >\n> > Signed-off-by: Brandon Williams \n> \n> The code is\n> Reviewed-by: Stefan Beller \n\n-- \nBrandon Williams\n"},{"id":"339956","messageId":"20180222193814.GA256918@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180222181400.GA19035@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-22T19:38:14Z","receivedAt":"2018-02-22T19:38:32Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff King wrote:\n> On Thu, Feb 22, 2018 at 10:07:15AM -0800, Brandon Williams wrote:\n>> On 02/22, Jeff King wrote:\n>>> On Wed, Feb 21, 2018 at 01:44:22PM -0800, Jonathan Tan wrote:\n>>>> On Tue, 6 Feb 2018 17:12:41 -0800\n>>>> Brandon Williams wrote:\n\n>>>>> In order to allow for code sharing with the server-side of fetch in\n>>>>> protocol-v2 convert upload-pack to be a builtin.\n[...]\n>>>> As Stefan mentioned in [1], also mention in the commit message that this\n>>>> means that the \"git-upload-pack\" invocation gains additional\n>>>> capabilities (for example, invoking a pager for --help).\n>>>\n>>> And possibly respecting pager.upload-pack, which would violate our rule\n>>> that it is safe to run upload-pack in untrusted repositories.\n>>\n>> And this isn't an issue with receive-pack because this same guarantee\n>> doesn't exist?\n>\n> Yes, exactly (which is confusing and weird, yes, but that's how it is).\n\nTo be clear, which of the following are you (most) worried about?\n\n 1. being invoked with --help and spawning a pager\n 2. receiving and acting on options between 'git' and 'upload-pack'\n 3. repository discovery\n 4. pager config\n 5. alias discovery\n 6. increased code surface / unknown threats\n\nFor (1), \"--help\" has to be the first argument. \"git daemon\" passes\n--strict so it doesn't happen there. \"git http-backend\" passes\n--stateless-rpc so it doesn't happen there. \"git shell\" sanitizes\ninput to avoid it happening there.\n\nA custom setup could provide their own entry point that doesn't do\nsuch sanitization. I think that in some sense it's out of our hands,\nbut it would be nice to harden as described upthread.\n\nFor (2), I am having trouble imagining a setup where it would happen.\n\nupload-pack doesn't have the RUN_SETUP or RUN_SETUP_GENTLY flag set,\nso (3) doesn't apply.\n\nAlthough in most setups the user does not control the config files on\na server, item (4) looks like a real issue worth solving. I think we\nshould introduce a flag to skip looking for pager config. We could\nuse it for receive-pack, too.\n\nBuiltins are handled before aliases, so (5) doesn't apply.\n\n(6) is a real issue: it is why \"git shell\" is not a builtin, for\nexample. But I would rather that we use appropriate sanitization\nbefore upload-pack is invoked than live in fear. git upload-pack is\nsufficiently complicated that I don't think the git.c wrapper\nincreases the complexity by a significant amount.\n\nThat leaves me with a personal answer of only being worried about (4)\nand not the rest. What do you think? Is one of the other items I\nlisted above worrisome, or is there another item I missed?\n\nThanks,\nJonathan\n"},{"id":"339957","messageId":"20180222193919.GN185096@google.com","threadId":"47524","inReplyTo":"CAGZ79kanWXGa3v4oJmoerh3XzisS+3V2oyO7WXK-tPTiHgSgxg@mail.gmail.com","subject":"Re: [PATCH v3 05/35] upload-pack: factor out processing lines","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T19:39:19Z","receivedAt":"2018-02-22T19:39:26Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Stefan Beller wrote:\n> On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> > Factor out the logic for processing shallow, deepen, deepen_since, and\n> > deepen_not lines into their own functions to simplify the\n> > 'receive_needs()' function in addition to making it easier to reuse some\n> > of this logic when implementing protocol_v2.\n> >\n> > Signed-off-by: Brandon Williams \n> \n> Reviewed-by: Stefan Beller \n> for the stated purpose of just refactoring existing code for better reuse later.\n> \n> I do have a few comments on the code in general,\n> which might be out of scope for this series.\n\nYeah you mentioned some comments in a previous round based on style\npreference. I'm going to refrain from changing the style of this patch\nsince it is a matter of preference.\n\n> \n> A close review would have been fastest if we had some sort of\n> https://public-inbox.org/git/20171025224620.27657-1-sbeller@google.com/\n> which I might revive soon for this purpose. (it showed that I would need it)\n> \n> \n> > + *depth = (int)strtol(arg, &end, 0);\n> \n> strtol is not used quite correctly here IMHO, as we do not\n> inspect errno for ERANGE\n> \n> Thanks,\n> Stefan\n\n-- \nBrandon Williams\n"},{"id":"339958","messageId":"20180222195133.GO185096@google.com","threadId":"47524","inReplyTo":"20180222113226.208f86fff80fb0d05c620433@google.com","subject":"Re: [PATCH v3 16/35] transport: convert transport_get_remote_refs to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-22T19:51:33Z","receivedAt":"2018-02-22T19:51:41Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Jonathan Tan wrote:\n> On Thu, 22 Feb 2018 10:26:47 -0800\n> Brandon Williams wrote:\n> \n> > On 02/21, Jonathan Tan wrote:\n> > > On Tue, 6 Feb 2018 17:12:53 -0800\n> > > Brandon Williams wrote:\n> > > \n> > > > -const struct ref *transport_get_remote_refs(struct transport *transport)\n> > > > +const struct ref *transport_get_remote_refs(struct transport *transport,\n> > > > +\t\t\t\t\t const struct argv_array *ref_patterns)\n> > > > {\n> > > > \tif (!transport->got_remote_refs) {\n> > > > -\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n> > > > +\t\ttransport->remote_refs =\n> > > > +\t\t\ttransport->vtable->get_refs_list(transport, 0,\n> > > > +\t\t\t\t\t\t\t ref_patterns);\n> > > > \t\ttransport->got_remote_refs = 1;\n> > > > \t}\n> > > \n> > > Should we do our own client-side filtering if the server side cannot do\n> > > it for us (because it doesn't support protocol v2)? Either way, this\n> > > decision should be mentioned in the commit message.\n> > \n> > If someone wants to add this in the future they can, but that is outside\n> > the scope of this series.\n> \n> In that case, also document that this function is allowed to return refs\n> that do not match the ref patterns.\n> \n> Unlike in patch 15 (which deals with the interface between the transport\n> code and transport vtables, which can be changed as long as the\n> transport code is aware of it, as I wrote in [1]), this may result in\n> user-visible differences depending on which protocol is used. But after\n> more thinking, I don't think we're in a situation yet where having extra\n> refs shown/written are harmful, and if it comes to that, we can tighten\n> this code later without backwards incompatibility. So, OK, this is fine.\n> \n> [1] https://public-inbox.org/git/20180221145639.c6cf2409ce2120109bdd169f@google.com/\n\nI'll add the documentation.\n\n-- \nBrandon Williams\n"},{"id":"339959","messageId":"CAGZ79ka_tYYhFE5OPfSgpuvssuivCTj7x3aXzLYLO22WGugTRA@mail.gmail.com","threadId":"47524","inReplyTo":"20180207011312.189834-8-bmwill@google.com","subject":"Re: [PATCH v3 07/35] connect: convert get_remote_heads to use struct packet_reader","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-22T19:52:54Z","receivedAt":"2018-02-22T19:53:01Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n\n> @@ -56,6 +62,41 @@ static void die_initial_contact(int unexpected)\n> \"and the repository exists.\"));\n> }\n>\n> +static enum protocol_version discover_version(struct packet_reader *reader)\n> +{\n> + enum protocol_version version = protocol_unknown_version;\n> +\n> + /*\n> + * Peek the first line of the server's response to\n> + * determine the protocol version the server is speaking.\n> + */\n> + switch (packet_reader_peek(reader)) {\n> + case PACKET_READ_EOF:\n> + die_initial_contact(0);\n> + case PACKET_READ_FLUSH:\n> + case PACKET_READ_DELIM:\n> + version = protocol_v0;\n> + break;\n> + case PACKET_READ_NORMAL:\n> + version = determine_protocol_version_client(reader->line);\n> + break;\n> + }\n> +\n> + /* Maybe process capabilities here, at least for v2 */\n\nWe do not (yet) react to v2, so this comment only makes\nsense after a later patch? If so please include it later,\nas this is confusing for now.\n\n\n> + switch (version) {\n> + case protocol_v1:\n> + /* Read the peeked version line */\n> + packet_reader_read(reader);\n\nI wonder if we want to assign version to v0 here,\nas now all v1 is done and we could treat the remaining\ncommunication as a v0. Not sure if that helps with some\nswitch/cases, but as we'd give all cases to have the compiler\nnot yell at us, this would be no big deal. So I guess we can keep\nit v1.\n\nWith or without the comment nit, this patch is\nReviewed-by: Stefan Beller \n"},{"id":"339962","messageId":"CAGZ79kbnioP+12xa2qfaYLyzdeCH8hh2Ri0xHD7rNciJiTYxmg@mail.gmail.com","threadId":"47524","inReplyTo":"20180207011312.189834-8-bmwill@google.com","subject":"Re: [PATCH v3 07/35] connect: convert get_remote_heads to use struct packet_reader","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-22T20:09:05Z","receivedAt":"2018-02-22T20:09:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> +static enum protocol_version discover_version(struct packet_reader *reader)\n> +{\n...\n> +\n> + /* Maybe process capabilities here, at least for v2 */\n> + switch (version) {\n> + case protocol_v1:\n> + /* Read the peeked version line */\n> + packet_reader_read(reader);\n> + break;\n> + case protocol_v0:\n> + break;\n> + case protocol_unknown_version:\n> + die(\"unknown protocol version: '%s'\\n\", reader->line);\n\nThe following patches introduce more of the switch(version) cases.\nAnd there it actually is a\n BUG(\"protocol version unknown? should have been set in discover_version\")\nbut here it is a mere\n die (_(\"The server uses a different protocol version than we can\nspeak: %s\\n\"),\n reader->line);\nso I would think here it is reasonable to add _(translation).\n"},{"id":"339967","messageId":"20180222201940.GB23561@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180222193814.GA256918@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T20:19:40Z","receivedAt":"2018-02-22T20:19:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 11:38:14AM -0800, Jonathan Nieder wrote:\n\n> >>> And possibly respecting pager.upload-pack, which would violate our rule\n> >>> that it is safe to run upload-pack in untrusted repositories.\n> >>\n> >> And this isn't an issue with receive-pack because this same guarantee\n> >> doesn't exist?\n> >\n> > Yes, exactly (which is confusing and weird, yes, but that's how it is).\n> \n> To be clear, which of the following are you (most) worried about?\n> \n> 1. being invoked with --help and spawning a pager\n> 2. receiving and acting on options between 'git' and 'upload-pack'\n> 3. repository discovery\n> 4. pager config\n> 5. alias discovery\n> 6. increased code surface / unknown threats\n\nMy immediate concern is (4). But my greater concern is that people who\nwork on git.c should not have to worry about accidentally violating this\nprinciple when they add a new feature or config option.\n\nIn other words, it seems like an accident waiting to happen. I'd be more\namenable to it if there was some compelling reason for it to be a\nbuiltin, but I don't see one listed in the commit message. I see only\n\"let's make it easier to share the code\", which AFAICT is equally served\nby just lib-ifying the code and calling it from the standalone\nupload-pack.c.\n\n> Although in most setups the user does not control the config files on\n> a server, item (4) looks like a real issue worth solving. I think we\n> should introduce a flag to skip looking for pager config. We could\n> use it for receive-pack, too.\n\nThere's not much point for receive-pack. It respects hooks, so any\nsecurity ship has already sailed there.\n\n-Peff\n"},{"id":"339969","messageId":"20180222202150.GA23985@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180222201940.GB23561@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T20:21:51Z","receivedAt":"2018-02-22T20:21:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 03:19:40PM -0500, Jeff King wrote:\n\n> > To be clear, which of the following are you (most) worried about?\n> > \n> > 1. being invoked with --help and spawning a pager\n> > 2. receiving and acting on options between 'git' and 'upload-pack'\n> > 3. repository discovery\n> > 4. pager config\n> > 5. alias discovery\n> > 6. increased code surface / unknown threats\n> \n> My immediate concern is (4). But my greater concern is that people who\n> work on git.c should not have to worry about accidentally violating this\n> principle when they add a new feature or config option.\n> \n> In other words, it seems like an accident waiting to happen. I'd be more\n> amenable to it if there was some compelling reason for it to be a\n> builtin, but I don't see one listed in the commit message. I see only\n> \"let's make it easier to share the code\", which AFAICT is equally served\n> by just lib-ifying the code and calling it from the standalone\n> upload-pack.c.\n\nBy the way, any decision here would presumably need to be extended to\ngit-serve, etc. The current property is that it's safe to fetch from an\nuntrusted repository, even over ssh. If we're keeping that for protocol\nv1, we'd want it to apply to protocol v2, as well.\n\n-Peff\n"},{"id":"339972","messageId":"CAGZ79kbrm2Ei=spoDbjrwVFoF+mi0AAsXNLDBgCozFVoXM6+1w@mail.gmail.com","threadId":"47524","inReplyTo":"20180207011312.189834-12-bmwill@google.com","subject":"Re: [PATCH v3 11/35] test-pkt-line: introduce a packet-line test helper","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-22T20:40:39Z","receivedAt":"2018-02-22T20:40:46Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n\n> +static void pack_line(const char *line)\n> +{\n> + if (!strcmp(line, \"0000\") || !strcmp(line, \"0000\\n\"))\n\nFrom our in-office discussion:\nv1/v0 packs pktlines twice in http, which is not possible to\nconstruct using this test helper when using the same string\nfor the packed and unpacked representation of flush and delim packets,\ni.e. test-pkt-line --pack $(test-pkt-line --pack 0000) would produce\n'0000' instead of '00090000\\n'.\nTo fix it we'd have to replace the unpacked versions of these pkts to\nsomething else such as \"FLUSH\" \"DELIM\".\n\nHowever as we do not anticipate the test helper to be used in further\ntests for v0, this ought to be no big issue.\nMaybe someone else cares though?\n"},{"id":"339976","messageId":"20180222212402.GB256918@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180222201940.GB23561@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-22T21:24:02Z","receivedAt":"2018-02-22T21:24:11Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff King wrote:\n> On Thu, Feb 22, 2018 at 11:38:14AM -0800, Jonathan Nieder wrote:\n\n>> To be clear, which of the following are you (most) worried about?\n>>\n>> 1. being invoked with --help and spawning a pager\n>> 2. receiving and acting on options between 'git' and 'upload-pack'\n>> 3. repository discovery\n>> 4. pager config\n>> 5. alias discovery\n>> 6. increased code surface / unknown threats\n>\n> My immediate concern is (4).\n\nThanks for clarifying.\n\n> But my greater concern is that people who\n> work on git.c should not have to worry about accidentally violating this\n> principle when they add a new feature or config option.\n\nThat sounds like a combination of (6) and insufficient documentation\nor tests. Ideas for how we can help prevent such accidents?\n\n> In other words, it seems like an accident waiting to happen. I'd be more\n> amenable to it if there was some compelling reason for it to be a\n> builtin, but I don't see one listed in the commit message. I see only\n> \"let's make it easier to share the code\", which AFAICT is equally served\n> by just lib-ifying the code and calling it from the standalone\n> upload-pack.c.\n\nIf we have so little control of the common code used by git commands\nthat could be invoked by a remote user, I think we're in trouble\nalready. I don't think being a builtin vs not makes that\nsignificantly different, since there are plenty of builtins that can\nbe triggered by remote users. Further, if we have so little control\nover the security properties of git.c, what hope do we have of making\nthe rest of libgit.a usable in secure code?\n\nIn other words, having to pay more attention to the git wrapper from a\nsecurity pov actually feels to me like a *good* thing. The git\nwrapper is the entry point to almost all git commands. If it is an\naccident waiting to happen, then anything that calls git commands is\nalready an accident waiting to happen. So how can we make it not an\naccident waiting to happen? :)\n\n>> Although in most setups the user does not control the config files on\n>> a server, item (4) looks like a real issue worth solving. I think we\n>> should introduce a flag to skip looking for pager config. We could\n>> use it for receive-pack, too.\n>\n> There's not much point for receive-pack. It respects hooks, so any\n> security ship has already sailed there.\n\nYet there are plenty of cases where people who can push are not\nsupposed to have root privilege. I am not worried about hooks\nspecifically (although the changes described at [1] might help and I\nstill plan to work on those) but I am worried about e.g. commandline\ninjection issues. I don't think we can treat receive-pack as out of\nscope.\n\nAnd to be clear, I don't think you were saying receive-pack *is* out\nof scope. But you seem to be trying to draw some boundary, where I\nonly see something fuzzier (e.g. if a bug only applies to\nreceive-pack, then that certainly decreases its impact, but it doesn't\nmake the impact go away).\n\nThanks,\nJonathan\n\n[1] https://public-inbox.org/git/20171002234517.GV19555@aiede.mtv.corp.google.com/\n"},{"id":"339977","messageId":"20180222212634.GC256918@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180222202150.GA23985@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-22T21:26:34Z","receivedAt":"2018-02-22T21:33:00Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff King wrote:\n\n> The current property is that it's safe to fetch from an\n> untrusted repository, even over ssh. If we're keeping that for protocol\n> v1, we'd want it to apply to protocol v2, as well.\n\nAh, this is what I had been missing (the non-ssh case).\n\nI see your point. I think we need to fix the pager config issue and add\nsome clarifying documentation to git.c so that people know what to look\nout for.\n\nKeep in mind that git upload-archive (a read-only command, just like\ngit upload-pack) also already has the same issues.\n\nThanks,\nJonathan\n"},{"id":"339979","messageId":"20180222214402.GA30638@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180222212402.GB256918@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T21:44:02Z","receivedAt":"2018-02-22T21:44:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 01:24:02PM -0800, Jonathan Nieder wrote:\n\n> > But my greater concern is that people who\n> > work on git.c should not have to worry about accidentally violating this\n> > principle when they add a new feature or config option.\n> \n> That sounds like a combination of (6) and insufficient documentation\n> or tests. Ideas for how we can help prevent such accidents?\n\nI don't think it's insufficient tests. How can we test against some\nproblem in the untrusted repository when the feature that would trigger\nit has not been written yet?\n\nE.g., imagine we begin to allow alias.* to override command names in\ngit.c. Now suddenly setting \"alias.upload-pack\" is a vulnerability.\nShould we add a test for that _now_ as a precaution? I don't think so,\nbecause we can't guess what new features are going to be added. So we'd\nbe lucky if such a test actually did anything useful.\n\nA comment could help, but it seems quite likely that whatever feature\nsomebody is adding might not be right next to that comment (and thus not\nseen). I think we're mostly relying on institutional knowledge during\nreview to catch these things. Which is not great, but I'm not sure where\nwe'd document that knowledge that people would actually see it at the\nright time.\n\n> > In other words, it seems like an accident waiting to happen. I'd be more\n> > amenable to it if there was some compelling reason for it to be a\n> > builtin, but I don't see one listed in the commit message. I see only\n> > \"let's make it easier to share the code\", which AFAICT is equally served\n> > by just lib-ifying the code and calling it from the standalone\n> > upload-pack.c.\n> \n> If we have so little control of the common code used by git commands\n> that could be invoked by a remote user, I think we're in trouble\n> already. I don't think being a builtin vs not makes that\n> significantly different, since there are plenty of builtins that can\n> be triggered by remote users. Further, if we have so little control\n> over the security properties of git.c, what hope do we have of making\n> the rest of libgit.a usable in secure code?\n\nI agree that the situation is already pretty dicey. But I also think\nthat using the git wrapper is more risky than the rest of libgit.a.\nThere's tons of dangerous code in libgit.a, but upload-pack is smart\nenough not to call it. And people modifying upload-pack have a greater\nchance of thinking about the security implications, because they know\nthey're working with upload-pack. Whereas people are likely to touch\ngit.c without considering upload-pack at all.\n\nThe big danger in libgit.a is from modifying some low-level code called\nby upload-pack in a way that trusts the on-disk contents more than it\nshould. My gut says that's less likely, though certainly not impossible\n(a likely candidate would perhaps be a ref backend config that opens up\nholes; e.g., if you could point a database backend at some random path).\n\n> In other words, having to pay more attention to the git wrapper from a\n> security pov actually feels to me like a *good* thing. The git\n> wrapper is the entry point to almost all git commands. If it is an\n> accident waiting to happen, then anything that calls git commands is\n> already an accident waiting to happen. So how can we make it not an\n> accident waiting to happen? :)\n\nBut I don't think it _is_ an accident waiting to happen for the rest of\nthe commands. upload-pack is special. The point is that people may touch\ngit.c thinking they are adding a nice new feature (like pager config, or\naliases, or default options, or whatever). And it _would_ be a nice new\nfeature for most commands, but not for upload-pack, because its\nrequirements are different.\n\nSo thinking about security in the git wrapper is just a burden for those\nother commands.\n\n> > There's not much point for receive-pack. It respects hooks, so any\n> > security ship has already sailed there.\n> \n> Yet there are plenty of cases where people who can push are not\n> supposed to have root privilege. I am not worried about hooks\n> specifically (although the changes described at [1] might help and I\n> still plan to work on those) but I am worried about e.g. commandline\n> injection issues. I don't think we can treat receive-pack as out of\n> scope.\n> \n> And to be clear, I don't think you were saying receive-pack *is* out\n> of scope. But you seem to be trying to draw some boundary, where I\n> only see something fuzzier (e.g. if a bug only applies to\n> receive-pack, then that certainly decreases its impact, but it doesn't\n> make the impact go away).\n\nRight, I think command-line injection is a separate issue. My concern is\n_just_ about \"can we be run against on-disk repo contents\". And nothing\nmatters for receive-pack there, because you can already execute\narbitrary code with hooks.\n\n-Peff\n"},{"id":"339980","messageId":"20180222214457.GB30638@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180222212634.GC256918@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T21:44:58Z","receivedAt":"2018-02-22T21:45:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 01:26:34PM -0800, Jonathan Nieder wrote:\n\n> Keep in mind that git upload-archive (a read-only command, just like\n> git upload-pack) also already has the same issues.\n\nYuck. I don't think we've ever made a historical promise about that. But\nthen, I don't think the promise about upload-pack has ever really been\ndocumented, except in mailing list discussions.\n\n-Peff\n"},{"id":"339981","messageId":"20180222135551.b9131397d4dc2b6bed376d0d@google.com","threadId":"47524","inReplyTo":"20180222185353.GH185096@google.com","subject":"Re: [PATCH v3 28/35] transport-helper: introduce stateless-connect","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-22T21:55:51Z","receivedAt":"2018-02-22T21:56:02Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 22 Feb 2018 10:53:53 -0800\nBrandon Williams wrote:\n\n> > > @@ -612,6 +615,11 @@ static int process_connect_service(struct transport *transport,\n> > > \tif (data->connect) {\n> > > \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n> > > \t\tret = run_connect(transport, &cmdbuf);\n> > > +\t} else if (data->stateless_connect) {\n> > > +\t\tstrbuf_addf(&cmdbuf, \"stateless-connect %s\\n\", name);\n> > > +\t\tret = run_connect(transport, &cmdbuf);\n> > > +\t\tif (ret)\n> > > +\t\t\ttransport->stateless_rpc = 1;\n> > \n> > Why is process_connect_service() falling back to stateless_connect if\n> > connect doesn't work? I don't think this fallback would work, as a\n> > client that needs \"connect\" might need its full capabilities.\n> \n> Right now there isn't really a notion of \"needing\" connect since if\n> connect fails then you need to fallback to doing the dumb thing. Also\n> note that there isn't all fallback from connect to stateless-connect\n> here. If the remote helper advertises connect, only connect will be\n> tried even if stateless-connect is advertised. So this only really\n> works in the case where stateless-connect is advertised and connect\n> isn't, as is with our http remote-helper.\n\nAfter some in-office discussion, I think I understand how this works.\nAssuming a HTTP server that supports protocol v2 (at least for\nls-refs/fetch):\n\n 1. Fetch, which supports protocol v2, will (indirectly) call\n process_connect_service. If it learns that it supports v2, it must\n know that what's returned may not be a fully bidirectional channel,\n but may only be a stateless-connect channel (and it does know).\n 2. Archive/upload-archive, which does not support protocol v2, will\n (indirectly) call process_connect_service. stateless_connect checks\n info/refs and observes that the server supports protocol v2, so it\n returns a stateless-connect channel. The user, being unaware of\n protocol versions, tries to use it, and it doesn't work. (This is a\n slight regression in that previously, it would fail more quickly -\n archive/upload-archive has always not supported HTTP because HTTP\n doesn't support connect.)\n\nI still think that it's too confusing for process_connect_service() to\nattempt to fallback to stateless-connect, at least because the user must\nremember that process_connect_service() returns such a channel if\nprotocol v2 is used (and existing code must be updated to know this).\nIt's probably better to have a new API that can return either a connect\nchannel or a stateless-connect channel, and the user will always use it\nas if it was a stateless-connect channel. The old API then can be\nseparately deprecated and removed, if desired.\n"},{"id":"339982","messageId":"20180222222146.GA30988@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180222214402.GA30638@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T22:21:46Z","receivedAt":"2018-02-22T22:21:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 04:44:02PM -0500, Jeff King wrote:\n\n> But I don't think it _is_ an accident waiting to happen for the rest of\n> the commands. upload-pack is special. The point is that people may touch\n> git.c thinking they are adding a nice new feature (like pager config, or\n> aliases, or default options, or whatever). And it _would_ be a nice new\n> feature for most commands, but not for upload-pack, because its\n> requirements are different.\n> \n> So thinking about security in the git wrapper is just a burden for those\n> other commands.\n\nAll of that said, I think the current code is quite dangerous already,\nand maybe even broken. upload-pack may run sub-commands like rev-list\nor pack-objects, which are themselves builtins.\n\nFor example:\n\n git init --bare evil.git\n git -C evil.git --work-tree=. commit --allow-empty -m foo\n git -C evil.git config pager.pack-objects 'echo >&2 oops'\n git clone --no-local evil.git victim\n\nThat doesn't _quite_ work, because we route pack-objects' stderr into a\npipe, which suppresses the pager. But we don't for rev-list, which we\ncall when checking reachability. It's a bit tricky to get a client to\ntrigger those for a vanilla fetch, though. Here's the best I could come\nup with:\n\n git init --bare evil.git\n git -C evil.git --work-tree=. commit --allow-empty -m one\n git -C evil.git config pager.rev-list 'echo >&2 oops'\n\n git init super\n (\n\tcd super\n\t# obviously use host:path if you're attacking somebody over ssh\n\tgit submodule add ../evil.git evil\n\tgit commit -am 'add evil submodule'\n )\n git -C evil.git config uploadpack.allowReachableSHA1InWant true\n git -C evil.git update-ref -d refs/heads/master\n\n git clone --recurse-submodules super victim\n\nI couldn't quite get it to work, but I think it's because I'm doing\nsomething wrong with the submodules. But I also think this attack would\n_have_ to be done over ssh, because on a local system the submodule\nclone would a hard-link rather than a real fetch.\n\n-Peff\n"},{"id":"339984","messageId":"20180222224235.GD256918@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180222222146.GA30988@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-22T22:42:35Z","receivedAt":"2018-02-22T22:42:43Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff King wrote:\n\n> All of that said, I think the current code is quite dangerous already,\n> and maybe even broken. upload-pack may run sub-commands like rev-list\n> or pack-objects, which are themselves builtins.\n\nSounds like more commands to set the IGNORE_PAGER_CONFIG flag for in\ngit.c.\n\nThanks for looking this over thoughtfully.\n\n[...]\n> I couldn't quite get it to work, but I think it's because I'm doing\n> something wrong with the submodules. But I also think this attack would\n> _have_ to be done over ssh, because on a local system the submodule\n> clone would a hard-link rather than a real fetch.\n\nWhat happens if the submodule URL starts with file://?\n\nThanks,\nJonathan\n"},{"id":"339986","messageId":"20180222230515.GA976@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180222224235.GD256918@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T23:05:16Z","receivedAt":"2018-02-22T23:05:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 02:42:35PM -0800, Jonathan Nieder wrote:\n\n> > I couldn't quite get it to work, but I think it's because I'm doing\n> > something wrong with the submodules. But I also think this attack would\n> > _have_ to be done over ssh, because on a local system the submodule\n> > clone would a hard-link rather than a real fetch.\n> \n> What happens if the submodule URL starts with file://?\n\nAh, that would do it. Or I guess any follow-up fetch.\n\nI'm still having trouble convincing submodules to fetch _just_ the\ndesired sha1, though. It always just fetches everything. I know there's\na way that this kicks in (that's why we have things like\nallowReachableSHA1InWant), but I'm not sufficiently well-versed in\nsubmodules to know how to trigger it.\n\n-Peff\n"},{"id":"339988","messageId":"20180222232326.GA8585@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180222230515.GA976@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-22T23:23:26Z","receivedAt":"2018-02-22T23:23:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 06:05:15PM -0500, Jeff King wrote:\n\n> On Thu, Feb 22, 2018 at 02:42:35PM -0800, Jonathan Nieder wrote:\n> \n> > > I couldn't quite get it to work, but I think it's because I'm doing\n> > > something wrong with the submodules. But I also think this attack would\n> > > _have_ to be done over ssh, because on a local system the submodule\n> > > clone would a hard-link rather than a real fetch.\n> > \n> > What happens if the submodule URL starts with file://?\n> \n> Ah, that would do it. Or I guess any follow-up fetch.\n> \n> I'm still having trouble convincing submodules to fetch _just_ the\n> desired sha1, though. It always just fetches everything. I know there's\n> a way that this kicks in (that's why we have things like\n> allowReachableSHA1InWant), but I'm not sufficiently well-versed in\n> submodules to know how to trigger it.\n\n This won't work anyway. I was right when I said that we don't\nredirect stderr for rev-list, but of course it's stdout that determines\nthe pager behavior. So I don't think you could get rev-list to trigger a\npager here.\n\nI don't think there's currently any vulnerability, but it's more to do\nwith luck than any amount of carefulness on our part.\n\n-Peff\n"},{"id":"339992","messageId":"20180223004514.GP185096@google.com","threadId":"47524","inReplyTo":"20180222094831.GB12442@sigill.intra.peff.net","subject":"Re: [PATCH v3 13/35] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-23T00:45:14Z","receivedAt":"2018-02-23T00:45:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Jeff King wrote:\n> On Tue, Feb 06, 2018 at 05:12:50PM -0800, Brandon Williams wrote:\n> \n> > +ls-refs takes in the following parameters wrapped in packet-lines:\n> > +\n> > + symrefs\n> > +\tIn addition to the object pointed by it, show the underlying ref\n> > +\tpointed by it when showing a symbolic ref.\n> > + peel\n> > +\tShow peeled tags.\n> > + ref-pattern \n> > +\tWhen specified, only references matching the one of the provided\n> > +\tpatterns are displayed.\n> \n> How do we match those patterns? That's probably an important thing to\n> include in the spec.\n\nYeah I thought about it when I first wrote it and was hoping that\nsomeone who nudge me in the right direction :)\n\n> \n> Looking at the code, I see:\n> \n> > +/*\n> > + * Check if one of the patterns matches the tail part of the ref.\n> > + * If no patterns were provided, all refs match.\n> > + */\n> > +static int ref_match(const struct argv_array *patterns, const char *refname)\n> \n> This kind of tail matching can't quite implement all of the current\n> behavior. Because we actually do the normal dwim_ref() matching, which\n> includes stuff like \"refs/remotes/%s/HEAD\".\n> \n> The other problem with tail-matching is that it's inefficient on the\n> server. Ideally we could get a request for \"master\" and only look up\n> refs/heads/master, refs/tags/master, etc. And if there are 50,000 refs\n> in refs/pull, we wouldn't have to process those at all. Of course this\n> is no worse than the current code, which not only looks at each ref but\n> actually _sends_ it. But it would be nice if we could fix this.\n> \n> There's some more discussion in this old thread:\n> \n> https://public-inbox.org/git/20161024132932.i42rqn2vlpocqmkq@sigill.intra.peff.net/\n\nThanks for the pointer. I was told to be wary a while about about\nperformance implications on the server but no discussion ensued till now\nabout it :)\n\nWe always have the ability to extend the patterns accepted via a feature\n(or capability) to ls-refs, so maybe the best thing to do now would only\nsupport a few patterns with specific semantics. Something like if you\nsay \"master\" only match against refs/heads/ and refs/tags/ and if you\nwant something else you would need to specify \"refs/pull/master\"?\n\nThat way we could only support globs at the end \"master*\" where * can\nmatch anything (including slashes)\n\n> \n> > +{\n> > +\tchar *pathbuf;\n> > +\tint i;\n> > +\n> > +\tif (!patterns->argc)\n> > +\t\treturn 1; /* no restriction */\n> > +\n> > +\tpathbuf = xstrfmt(\"/%s\", refname);\n> > +\tfor (i = 0; i < patterns->argc; i++) {\n> > +\t\tif (!wildmatch(patterns->argv[i], pathbuf, 0)) {\n> > +\t\t\tfree(pathbuf);\n> > +\t\t\treturn 1;\n> > +\t\t}\n> > +\t}\n> > +\tfree(pathbuf);\n> > +\treturn 0;\n> > +}\n> \n> Does the client have to be aware that we're using wildmatch? I think\n> they'd need \"refs/heads/**\" to actually implement what we usually\n> specify in refspecs as \"refs/heads/*\". Or does the lack of WM_PATHNAME\n> make this work with just \"*\"?\n> \n> Do we anticipate that the client would left-anchor the refspec like\n> \"/refs/heads/*\" so that in theory the server could avoid looking outside\n> of /refs/heads/?\n\nYeah we may want to anchor it by providing the leading '/' instead of\njust \"refs/\".\n\n> \n> -Peff\n\nI need to read over the discussion you linked to more but what sort of\nref patterns do you believe we should support as part of the initial\nrelease of v2? It seems like you wanted this at some point in the past\nso I assume you have an idea of what sort of filtering would be\nbeneficial.\n\n-- \nBrandon Williams\n"},{"id":"340057","messageId":"20180223113722.ac4a8d3ef18bec31f20e79f7@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-24-bmwill@google.com","subject":"Re: [PATCH v3 23/35] fetch-pack: support shallow requests","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-23T19:37:22Z","receivedAt":"2018-02-23T19:37:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:13:00 -0800\nBrandon Williams wrote:\n\n> @@ -1090,6 +1110,10 @@ static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n> \tif (prefer_ofs_delta)\n> \t\tpacket_buf_write(&req_buf, \"ofs-delta\");\n> \n> +\t/* Add shallow-info and deepen request */\n> +\tif (server_supports_feature(\"fetch\", \"shallow\", 1))\n> +\t\tadd_shallow_requests(&req_buf, args);\n\nOne more thing I observed when trying to implement the server side in\nJGit - the last argument should be 0, not 1, right? I don't think that\n\"shallow\" should be required on the server unless we really need it.\n"},{"id":"340063","messageId":"20180223195605.GA234838@google.com","threadId":"47524","inReplyTo":"20180223113722.ac4a8d3ef18bec31f20e79f7@google.com","subject":"Re: [PATCH v3 23/35] fetch-pack: support shallow requests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-23T19:56:05Z","receivedAt":"2018-02-23T19:56:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/23, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:13:00 -0800\n> Brandon Williams wrote:\n> \n> > @@ -1090,6 +1110,10 @@ static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n> > \tif (prefer_ofs_delta)\n> > \t\tpacket_buf_write(&req_buf, \"ofs-delta\");\n> > \n> > +\t/* Add shallow-info and deepen request */\n> > +\tif (server_supports_feature(\"fetch\", \"shallow\", 1))\n> > +\t\tadd_shallow_requests(&req_buf, args);\n> \n> One more thing I observed when trying to implement the server side in\n> JGit - the last argument should be 0, not 1, right? I don't think that\n> \"shallow\" should be required on the server unless we really need it.\n\nGood catch, I'll fix that.\n\n-- \nBrandon Williams\n"},{"id":"340072","messageId":"20180223210904.GB234838@google.com","threadId":"47524","inReplyTo":"20180222202150.GA23985@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-23T21:09:04Z","receivedAt":"2018-02-23T21:09:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Jeff King wrote:\n> On Thu, Feb 22, 2018 at 03:19:40PM -0500, Jeff King wrote:\n> \n> > > To be clear, which of the following are you (most) worried about?\n> > > \n> > > 1. being invoked with --help and spawning a pager\n> > > 2. receiving and acting on options between 'git' and 'upload-pack'\n> > > 3. repository discovery\n> > > 4. pager config\n> > > 5. alias discovery\n> > > 6. increased code surface / unknown threats\n> > \n> > My immediate concern is (4). But my greater concern is that people who\n> > work on git.c should not have to worry about accidentally violating this\n> > principle when they add a new feature or config option.\n> > \n> > In other words, it seems like an accident waiting to happen. I'd be more\n> > amenable to it if there was some compelling reason for it to be a\n> > builtin, but I don't see one listed in the commit message. I see only\n> > \"let's make it easier to share the code\", which AFAICT is equally served\n> > by just lib-ifying the code and calling it from the standalone\n> > upload-pack.c.\n> \n> By the way, any decision here would presumably need to be extended to\n> git-serve, etc. The current property is that it's safe to fetch from an\n> untrusted repository, even over ssh. If we're keeping that for protocol\n> v1, we'd want it to apply to protocol v2, as well.\n> \n> -Peff\n\nThis may be more complicated. Right now (for backward compatibility)\nall fetches for v2 are issued to the upload-pack endpoint. So even\nthough I've introduced git-serve it doesn't have requests issued to it\nand no requests can be issued to it currently (support isn't added to\nhttp-backend or git-daemon). This just means that the command already\nexists to make it easy for testing specific v2 stuff and if we want to\nexpose it as an endpoint (like when we have a brand new server command\nthat is completely incompatible with v1) its already there and support\njust needs to be plumbed in.\n\nThis whole notion of treating upload-pack differently from receive-pack\nhas bad consequences for v2 though. The idea for v2 is to be able to\nrun any number of commands via the same endpoint, so at the end of the\nday the endpoint you used is irrelevant. So you could issue both fetch\nand push commands via the same endpoint in v2 whether its git-serve,\nreceive-pack, or upload-pack. So really, like Jonathan has said\nelsewhere, we need to figure out how to be ok with having receive-pack\nand upload-pack builtins, or having neither of them builtins, because it\ndoesn't make much sense for v2 to straddle that line. I mean you could\ndo some complicated advertising of commands based on the endpoint you\nhit, but then what does that mean if you're hitting the git-serve\nendpoint where you should presumably be able to do any operation.\n\n-- \nBrandon Williams\n"},{"id":"340075","messageId":"20180223212231.GC234838@google.com","threadId":"47524","inReplyTo":"CAGZ79kbrm2Ei=spoDbjrwVFoF+mi0AAsXNLDBgCozFVoXM6+1w@mail.gmail.com","subject":"Re: [PATCH v3 11/35] test-pkt-line: introduce a packet-line test helper","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-23T21:22:31Z","receivedAt":"2018-02-23T21:22:39Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Stefan Beller wrote:\n> On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> \n> > +static void pack_line(const char *line)\n> > +{\n> > + if (!strcmp(line, \"0000\") || !strcmp(line, \"0000\\n\"))\n> \n> From our in-office discussion:\n> v1/v0 packs pktlines twice in http, which is not possible to\n> construct using this test helper when using the same string\n> for the packed and unpacked representation of flush and delim packets,\n> i.e. test-pkt-line --pack $(test-pkt-line --pack 0000) would produce\n> '0000' instead of '00090000\\n'.\n> To fix it we'd have to replace the unpacked versions of these pkts to\n> something else such as \"FLUSH\" \"DELIM\".\n> \n> However as we do not anticipate the test helper to be used in further\n> tests for v0, this ought to be no big issue.\n> Maybe someone else cares though?\n\nI'm going to punt and say, if someone cares enough they can update this\ntest-helper when they want to use it for v1/v0 stuff.\n\n-- \nBrandon Williams\n"},{"id":"340077","messageId":"20180223213000.GD234838@google.com","threadId":"47524","inReplyTo":"CAGZ79kbnioP+12xa2qfaYLyzdeCH8hh2Ri0xHD7rNciJiTYxmg@mail.gmail.com","subject":"Re: [PATCH v3 07/35] connect: convert get_remote_heads to use struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-23T21:30:00Z","receivedAt":"2018-02-23T21:30:08Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Stefan Beller wrote:\n> > +static enum protocol_version discover_version(struct packet_reader *reader)\n> > +{\n> ...\n> > +\n> > + /* Maybe process capabilities here, at least for v2 */\n> > + switch (version) {\n> > + case protocol_v1:\n> > + /* Read the peeked version line */\n> > + packet_reader_read(reader);\n> > + break;\n> > + case protocol_v0:\n> > + break;\n> > + case protocol_unknown_version:\n> > + die(\"unknown protocol version: '%s'\\n\", reader->line);\n> \n> The following patches introduce more of the switch(version) cases.\n> And there it actually is a\n> BUG(\"protocol version unknown? should have been set in discover_version\")\n> but here it is a mere\n> die (_(\"The server uses a different protocol version than we can\n> speak: %s\\n\"),\n> reader->line);\n> so I would think here it is reasonable to add _(translation).\n\nThis should be a BUG as it shouldn't ever be unknown at this point. And\nI'll also drop that comment.\n\n-- \nBrandon Williams\n"},{"id":"340078","messageId":"20180223213315.GE234838@google.com","threadId":"47524","inReplyTo":"20180221144515.d180a26627a33168b19e0a4f@google.com","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-23T21:33:15Z","receivedAt":"2018-02-23T21:33:23Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/21, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:12:49 -0800\n> Brandon Williams wrote:\n> \n> > .gitignore | 1 +\n> > Documentation/technical/protocol-v2.txt | 114 +++++++++++++++\n> > Makefile | 2 +\n> > builtin.h | 1 +\n> > builtin/serve.c | 30 ++++\n> > git.c | 1 +\n> > serve.c | 250 ++++++++++++++++++++++++++++++++\n> > serve.h | 15 ++\n> > t/t5701-git-serve.sh | 60 ++++++++\n> > 9 files changed, 474 insertions(+)\n> > create mode 100644 Documentation/technical/protocol-v2.txt\n> > create mode 100644 builtin/serve.c\n> > create mode 100644 serve.c\n> > create mode 100644 serve.h\n> > create mode 100755 t/t5701-git-serve.sh\n> \n> As someone who is implementing the server side of protocol V2 in JGit, I\n> now have a bit more insight into this :-)\n> \n> First of all, I used to not have a strong opinion on the existence of a\n> new endpoint, but now I think that it's better to *not* have git-serve.\n> As it is, as far as I can tell, upload-pack also needs to support (and\n> does support, as of the end of this patch set) protocol v2 anyway, so it\n> might be better to merely upgrade upload-pack.\n\nHaving it allows for easier testing and the easy ability to make it a\ntrue endpoint when we want to. As of right now, git-serve isn't an\nendpoint as you can't issue requests there via http-backend or\ngit-daemon.\n\n> \n> > +A client then responds to select the command it wants with any particular\n> > +capabilities or arguments. There is then an optional section where the\n> > +client can provide any command specific parameters or queries.\n> > +\n> > + command-request = command\n> > +\t\t capability-list\n> > +\t\t (command-args)\n> \n> If you are stating that this is optional, write \"*1command-args\". (RFC\n> 5234 also supports square brackets, but \"*1\" is already used in\n> pack-protocol.txt and http-protocol.txt.)\n> \n> > +\t\t flush-pkt\n> > + command = PKT-LINE(\"command=\" key LF)\n> > + command-args = delim-pkt\n> > +\t\t *arg\n> > + arg = 1*CHAR\n> \n> arg should be wrapped in PKT-LINE, I think, and terminated by an LF.\n\n-- \nBrandon Williams\n"},{"id":"340080","messageId":"20180223214557.GF234838@google.com","threadId":"47524","inReplyTo":"20180222093327.GA12442@sigill.intra.peff.net","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-23T21:45:57Z","receivedAt":"2018-02-23T21:46:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Jeff King wrote:\n> On Tue, Feb 06, 2018 at 05:12:49PM -0800, Brandon Williams wrote:\n> \n> > +In protocol v2 communication is command oriented. When first contacting a\n> > +server a list of capabilities will advertised. Some of these capabilities\n> > +will be commands which a client can request be executed. Once a command\n> > +has completed, a client can reuse the connection and request that other\n> > +commands be executed.\n> \n> If I understand this correctly, we'll potentially have a lot more\n> round-trips between the client and server (one per \"command\"). And for\n> git-over-http, each one will be its own HTTP request?\n> \n> We've traditionally tried to minimize HTTP requests, but I guess it's\n> not too bad if we can keep the connection open in most cases. Then we\n> just suffer some extra framing bytes, but we don't have to re-establish\n> the TCP connection each time.\n> \n> I do wonder if the extra round trips will be noticeable in high-latency\n> conditions. E.g., if I'm 200ms away, converting the current\n> ref-advertisement spew to \"capabilities, then the client asks for refs,\n> then we spew the refs\" is going to cost an extra 200ms, even if the\n> fetch just ends up being a noop. I'm not sure how bad that is in the\n> grand scheme of things (after all, the TCP handshake involves some\n> round-trips, too).\n\nI think this is the price of extending the protocol in a backward\ncompatible way. If we don't want to be backwards compatible (allowing\nfor graceful fallback to v1) then we could design this differently.\nEven so we're not completely out of luck just yet.\n\nBack when I introduced the GIT_PROTOCOL side-channel I was able to\ndemonstrate that arbitrary data could be sent to the server and it would\nonly respect the stuff it knows about. This means that we can do a\nfollow up to v2 at some point to introduce an optimization where we can\nstuff a request into GIT_PROTOCOL and short-circuit the first round-trip\nif the server supports it.\n\n> \n> > + Capability Advertisement\n> > +--------------------------\n> > +\n> > +A server which decides to communicate (based on a request from a client)\n> > +using protocol version 2, notifies the client by sending a version string\n> > +in its initial response followed by an advertisement of its capabilities.\n> > +Each capability is a key with an optional value. Clients must ignore all\n> > +unknown keys. Semantics of unknown values are left to the definition of\n> > +each key. Some capabilities will describe commands which can be requested\n> > +to be executed by the client.\n> > +\n> > + capability-advertisement = protocol-version\n> > +\t\t\t capability-list\n> > +\t\t\t flush-pkt\n> > +\n> > + protocol-version = PKT-LINE(\"version 2\" LF)\n> > + capability-list = *capability\n> > + capability = PKT-LINE(key[=value] LF)\n> > +\n> > + key = 1*CHAR\n> > + value = 1*CHAR\n> > + CHAR = 1*(ALPHA / DIGIT / \"-\" / \"_\")\n> > +\n> > +A client then responds to select the command it wants with any particular\n> > +capabilities or arguments. There is then an optional section where the\n> > +client can provide any command specific parameters or queries.\n> > +\n> > + command-request = command\n> > +\t\t capability-list\n> > +\t\t (command-args)\n> > +\t\t flush-pkt\n> > + command = PKT-LINE(\"command=\" key LF)\n> > + command-args = delim-pkt\n> > +\t\t *arg\n> > + arg = 1*CHAR\n> \n> For a single stateful TCP connection like git:// or git-over-ssh, the\n> client would get the capabilities once and then issue a series of\n> commands. For git-over-http, how does it work?\n> \n> The client speaks first in HTTP, so we'd first make a request to get\n> just the capabilities from the server? And then proceed from there with\n> a series of requests, assuming that the capabilities for each server we\n> subsequently contact are the same? That's probably reasonable (and\n> certainly the existing http protocol makes that capabilities\n> assumption).\n> \n> I don't see any documentation on how this all works with http. But\n\nI can add in a bit for the initial request when using http, but the rest\nof it should function the same.\n\n> reading patch 34, it looks like we just do the usual\n> service=git-upload-pack request (with the magic request for v2), and\n> then the server would send us capabilities. Which follows my line of\n> thinking in the paragraph above.\n\nYes this is exactly how it should work. First we make an info/refs\nrequest and if the server speaks v2 then instead of a refs request we\nshould get back a capability listing. Then subsequent requests are made\nassuming the capabilities are the same like we've done with the\nexisting protocol.\n\nThe great thing about this is that from the POV of the git-client, it\ndoesn't care if its speaking using the git://, ssh://, file://, or\nhttp:// transport; it's all the same protocol. In my next re-roll I'll\neven drop the \"# service\" bit from the http server response and then the\nresponses will truly be identical in all cases.\n\n-- \nBrandon Williams\n"},{"id":"340081","messageId":"CAGZ79kbh0q8=N07yc+gGnLjggco_+H4YiYAiNOXR3yvO6QX6uw@mail.gmail.com","threadId":"47524","inReplyTo":"20180223213000.GD234838@google.com","subject":"Re: [PATCH v3 07/35] connect: convert get_remote_heads to use struct packet_reader","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-23T21:48:31Z","receivedAt":"2018-02-23T21:48:37Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Feb 23, 2018 at 1:30 PM, Brandon Williams wrote:\n> On 02/22, Stefan Beller wrote:\n>> > +static enum protocol_version discover_version(struct packet_reader *reader)\n>> > +{\n>> ...\n>> > +\n>> > + /* Maybe process capabilities here, at least for v2 */\n>> > + switch (version) {\n>> > + case protocol_v1:\n>> > + /* Read the peeked version line */\n>> > + packet_reader_read(reader);\n>> > + break;\n>> > + case protocol_v0:\n>> > + break;\n>> > + case protocol_unknown_version:\n>> > + die(\"unknown protocol version: '%s'\\n\", reader->line);\n>>\n>> The following patches introduce more of the switch(version) cases.\n>> And there it actually is a\n>> BUG(\"protocol version unknown? should have been set in discover_version\")\n>> but here it is a mere\n>> die (_(\"The server uses a different protocol version than we can\n>> speak: %s\\n\"),\n>> reader->line);\n>> so I would think here it is reasonable to add _(translation).\n>\n> This should be a BUG as it shouldn't ever be unknown at this point. And\n> I'll also drop that comment.\n\nHuh?\nThen I miss-understood the flow of code. When the server announces its\nanswer is version 42, but the client cannot handle it, which die call is\nresponsible for reporting it to the user?\n(That is technically a BUG on the server side, as we probably never\nasked for v42, so I would not want to print BUG locally at the client?)\n"},{"id":"340094","messageId":"20180223225637.GG234838@google.com","threadId":"47524","inReplyTo":"CAGZ79kbh0q8=N07yc+gGnLjggco_+H4YiYAiNOXR3yvO6QX6uw@mail.gmail.com","subject":"Re: [PATCH v3 07/35] connect: convert get_remote_heads to use struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-23T22:56:37Z","receivedAt":"2018-02-23T22:56:55Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/23, Stefan Beller wrote:\n> On Fri, Feb 23, 2018 at 1:30 PM, Brandon Williams wrote:\n> > On 02/22, Stefan Beller wrote:\n> >> > +static enum protocol_version discover_version(struct packet_reader *reader)\n> >> > +{\n> >> ...\n> >> > +\n> >> > + /* Maybe process capabilities here, at least for v2 */\n> >> > + switch (version) {\n> >> > + case protocol_v1:\n> >> > + /* Read the peeked version line */\n> >> > + packet_reader_read(reader);\n> >> > + break;\n> >> > + case protocol_v0:\n> >> > + break;\n> >> > + case protocol_unknown_version:\n> >> > + die(\"unknown protocol version: '%s'\\n\", reader->line);\n> >>\n> >> The following patches introduce more of the switch(version) cases.\n> >> And there it actually is a\n> >> BUG(\"protocol version unknown? should have been set in discover_version\")\n> >> but here it is a mere\n> >> die (_(\"The server uses a different protocol version than we can\n> >> speak: %s\\n\"),\n> >> reader->line);\n> >> so I would think here it is reasonable to add _(translation).\n> >\n> > This should be a BUG as it shouldn't ever be unknown at this point. And\n> > I'll also drop that comment.\n> \n> Huh?\n> Then I miss-understood the flow of code. When the server announces its\n> answer is version 42, but the client cannot handle it, which die call is\n> responsible for reporting it to the user?\n> (That is technically a BUG on the server side, as we probably never\n> asked for v42, so I would not want to print BUG locally at the client?)\n\nThis is handled in \n`determine_protocol_version_client(const char *server_response)`,\nwhich is just a few lines out of context here.\n\n-- \nBrandon Williams\n"},{"id":"340112","messageId":"20180224001954.GA153423@google.com","threadId":"47524","inReplyTo":"20180223004514.GP185096@google.com","subject":"Re: [PATCH v3 13/35] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-24T00:19:54Z","receivedAt":"2018-02-24T00:20:03Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/22, Brandon Williams wrote:\n> On 02/22, Jeff King wrote:\n> > On Tue, Feb 06, 2018 at 05:12:50PM -0800, Brandon Williams wrote:\n> > \n> > > +ls-refs takes in the following parameters wrapped in packet-lines:\n> > > +\n> > > + symrefs\n> > > +\tIn addition to the object pointed by it, show the underlying ref\n> > > +\tpointed by it when showing a symbolic ref.\n> > > + peel\n> > > +\tShow peeled tags.\n> > > + ref-pattern \n> > > +\tWhen specified, only references matching the one of the provided\n> > > +\tpatterns are displayed.\n> > \n> > How do we match those patterns? That's probably an important thing to\n> > include in the spec.\n> \n> Yeah I thought about it when I first wrote it and was hoping that\n> someone who nudge me in the right direction :)\n> \n> > \n> > Looking at the code, I see:\n> > \n> > > +/*\n> > > + * Check if one of the patterns matches the tail part of the ref.\n> > > + * If no patterns were provided, all refs match.\n> > > + */\n> > > +static int ref_match(const struct argv_array *patterns, const char *refname)\n> > \n> > This kind of tail matching can't quite implement all of the current\n> > behavior. Because we actually do the normal dwim_ref() matching, which\n> > includes stuff like \"refs/remotes/%s/HEAD\".\n> > \n> > The other problem with tail-matching is that it's inefficient on the\n> > server. Ideally we could get a request for \"master\" and only look up\n> > refs/heads/master, refs/tags/master, etc. And if there are 50,000 refs\n> > in refs/pull, we wouldn't have to process those at all. Of course this\n> > is no worse than the current code, which not only looks at each ref but\n> > actually _sends_ it. But it would be nice if we could fix this.\n> > \n> > There's some more discussion in this old thread:\n> > \n> > https://public-inbox.org/git/20161024132932.i42rqn2vlpocqmkq@sigill.intra.peff.net/\n> \n> Thanks for the pointer. I was told to be wary a while about about\n> performance implications on the server but no discussion ensued till now\n> about it :)\n> \n> We always have the ability to extend the patterns accepted via a feature\n> (or capability) to ls-refs, so maybe the best thing to do now would only\n> support a few patterns with specific semantics. Something like if you\n> say \"master\" only match against refs/heads/ and refs/tags/ and if you\n> want something else you would need to specify \"refs/pull/master\"?\n> \n> That way we could only support globs at the end \"master*\" where * can\n> match anything (including slashes)\n\nAfter some in-office discussion it seems like the best thing to do for\nthis (right now since if we change our mind we can just introduce a\ncapability which extends the patterns supported) would be to left-anchor\nthe ref-patterns and only allow for a single wildcard character '*'\nwhich matches zero or more characters (and doesn't care about slashes\n'/'). This wildcard character should only be supported at the end of\nthe ref pattern. This means that if a client wants 'master' then they\nwould need to specify 'refs/heads/master' (and the other\nref_rev_parse_rules expansions) as a ref pattern. But they could say\n\"refs/heads/*\" for all refs under refs/heads.\n\n> \n> > \n> > > +{\n> > > +\tchar *pathbuf;\n> > > +\tint i;\n> > > +\n> > > +\tif (!patterns->argc)\n> > > +\t\treturn 1; /* no restriction */\n> > > +\n> > > +\tpathbuf = xstrfmt(\"/%s\", refname);\n> > > +\tfor (i = 0; i < patterns->argc; i++) {\n> > > +\t\tif (!wildmatch(patterns->argv[i], pathbuf, 0)) {\n> > > +\t\t\tfree(pathbuf);\n> > > +\t\t\treturn 1;\n> > > +\t\t}\n> > > +\t}\n> > > +\tfree(pathbuf);\n> > > +\treturn 0;\n> > > +}\n> > \n> > Does the client have to be aware that we're using wildmatch? I think\n> > they'd need \"refs/heads/**\" to actually implement what we usually\n> > specify in refspecs as \"refs/heads/*\". Or does the lack of WM_PATHNAME\n> > make this work with just \"*\"?\n> > \n> > Do we anticipate that the client would left-anchor the refspec like\n> > \"/refs/heads/*\" so that in theory the server could avoid looking outside\n> > of /refs/heads/?\n> \n> Yeah we may want to anchor it by providing the leading '/' instead of\n> just \"refs/\".\n> \n> > \n> > -Peff\n> \n> I need to read over the discussion you linked to more but what sort of\n> ref patterns do you believe we should support as part of the initial\n> release of v2? It seems like you wanted this at some point in the past\n> so I assume you have an idea of what sort of filtering would be\n> beneficial.\n> \n> -- \n> Brandon Williams\n\n-- \nBrandon Williams\n"},{"id":"340143","messageId":"20180223165407.2d2f4343b2497172523f9f66@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-22-bmwill@google.com","subject":"Re: [PATCH v3 21/35] fetch-pack: perform a fetch using v2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-24T00:54:07Z","receivedAt":"2018-02-24T00:54:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 6 Feb 2018 17:12:58 -0800\nBrandon Williams wrote:\n\n> +\twhile ((oid = get_rev())) {\n> +\t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n> +\t\tif (++haves_added >= INITIAL_FLUSH)\n> +\t\t\tbreak;\n> +\t};\n\nUnnecessary semicolon after closing brace.\n\n> +\t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> +\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n> +\t\t\t\tstate = FETCH_DONE;\n> +\t\t\telse\n> +\t\t\t\tstate = FETCH_SEND_REQUEST;\n> +\t\t\tbreak;\n\nI haven't looked at this patch in detail, but I found a bug that can be\nreproduced if you patch the following onto this patch:\n\n --- a/t/t5702-protocol-v2.sh\n +++ b/t/t5702-protocol-v2.sh\n @@ -124,6 +124,7 @@ test_expect_success 'clone with file:// using protocol v2' '\n \n test_expect_success 'fetch with file:// using protocol v2' '\n test_commit -C file_parent two &&\n + git -C file_parent tag -d one &&\n \n GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n fetch origin 2>log &&\n @@ -133,7 +134,8 @@ test_expect_success 'fetch with file:// using protocol v2' '\n test_cmp expect actual &&\n \n # Server responded using protocol v2\n - grep \"fetch< version 2\" log\n + grep \"fetch< version 2\" log &&\n + grep \"have \" log\n '\n\nMerely including the second hunk (the one with 'grep \"have \"') does not\nmake the test fail, but including both the first and second hunks does.\nThat is, fetch v2 emits \"have\" only for remote refs that point to\nobjects we already have, not for local refs.\n\nEverything still appears to work, except that packfiles are usually much\nlarger than they need to be.\n\nI did some digging in the code and found out that the equivalent of\nfind_common() (which calls `for_each_ref(rev_list_insert_ref_oid,\nNULL)`) was not called in v2. In v1, find_common() is called immediately\nafter everything_local(), but there is no equivalent in v2. (I quoted\nthe invocation of everything_local() in v2 above.)\n"},{"id":"340156","messageId":"20180224040149.GA16743@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180223004514.GP185096@google.com","subject":"Re: [PATCH v3 13/35] ls-refs: introduce ls-refs server command","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-24T04:01:49Z","receivedAt":"2018-02-24T04:01:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 22, 2018 at 04:45:14PM -0800, Brandon Williams wrote:\n\n> > This kind of tail matching can't quite implement all of the current\n> > behavior. Because we actually do the normal dwim_ref() matching, which\n> > includes stuff like \"refs/remotes/%s/HEAD\".\n> > \n> > The other problem with tail-matching is that it's inefficient on the\n> > server. Ideally we could get a request for \"master\" and only look up\n> > refs/heads/master, refs/tags/master, etc. And if there are 50,000 refs\n> > in refs/pull, we wouldn't have to process those at all. Of course this\n> > is no worse than the current code, which not only looks at each ref but\n> > actually _sends_ it. But it would be nice if we could fix this.\n> > \n> > There's some more discussion in this old thread:\n> > \n> > https://public-inbox.org/git/20161024132932.i42rqn2vlpocqmkq@sigill.intra.peff.net/\n> \n> Thanks for the pointer. I was told to be wary a while about about\n> performance implications on the server but no discussion ensued till now\n> about it :)\n> \n> We always have the ability to extend the patterns accepted via a feature\n> (or capability) to ls-refs, so maybe the best thing to do now would only\n> support a few patterns with specific semantics. Something like if you\n> say \"master\" only match against refs/heads/ and refs/tags/ and if you\n> want something else you would need to specify \"refs/pull/master\"?\n\nThe big question is whether you want to break compatibility with the\nexisting program behavior. If not, then I think you have to ask for\nevery variant in ref_rev_parse_rules (of which there are 6 variants).\n\nWhich sounds pretty gross, but it actually may not be _too_ bad. Most\nfetches tend to ask for either a single name, or they use left-anchored\nwildcards. So it would work to just have the client expand all of the\npossibilities itself into fully-qualified refs, and keep the server as\ndumb as possible.\n\nAnd then the server for now can just cull based on the pattern list,\nlike you have here. But later, we could optimize it to look up the\nindividual patterns, which should be cheaper, since we'd generally have\nmany fewer patterns than total refs.\n\n> > Does the client have to be aware that we're using wildmatch? I think\n> > they'd need \"refs/heads/**\" to actually implement what we usually\n> > specify in refspecs as \"refs/heads/*\". Or does the lack of WM_PATHNAME\n> > make this work with just \"*\"?\n> > \n> > Do we anticipate that the client would left-anchor the refspec like\n> > \"/refs/heads/*\" so that in theory the server could avoid looking outside\n> > of /refs/heads/?\n> \n> Yeah we may want to anchor it by providing the leading '/' instead of\n> just \"refs/\".\n\nI actually wonder if we should just specify that the patterns must\n_always_ be fully-qualified, but may end with a single \"/*\" to iterate\nover wildcards. Or even simpler, that \"refs/heads/foo\" would find that\nref itself, and anything under it.\n\nThat drops any question about how wildcards work (e.g., does \"refs/foo*\"\nwork to find \"refs/foobar\"?).\n\n> I need to read over the discussion you linked to more but what sort of\n> ref patterns do you believe we should support as part of the initial\n> release of v2? It seems like you wanted this at some point in the past\n> so I assume you have an idea of what sort of filtering would be\n> beneficial.\n\nMy goals were just optimizing:\n\n 1. Don't send all the refs across the wire if we can avoid it.\n\n 2. Don't even iterate over all the refs internally if we can avoid it.\n\nEspecially with the new binary-searching packed-refs code, we should be\nable to serve a request like \"ls-refs refs/heads/*\" without looking into\n\"refs/pull\" or \"refs/changes\" at all.\n\n-Peff\n"},{"id":"340157","messageId":"20180224040350.GB16743@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180224001954.GA153423@google.com","subject":"Re: [PATCH v3 13/35] ls-refs: introduce ls-refs server command","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-24T04:03:50Z","receivedAt":"2018-02-24T04:03:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 23, 2018 at 04:19:54PM -0800, Brandon Williams wrote:\n\n> > We always have the ability to extend the patterns accepted via a feature\n> > (or capability) to ls-refs, so maybe the best thing to do now would only\n> > support a few patterns with specific semantics. Something like if you\n> > say \"master\" only match against refs/heads/ and refs/tags/ and if you\n> > want something else you would need to specify \"refs/pull/master\"?\n> > \n> > That way we could only support globs at the end \"master*\" where * can\n> > match anything (including slashes)\n> \n> After some in-office discussion it seems like the best thing to do for\n> this (right now since if we change our mind we can just introduce a\n> capability which extends the patterns supported) would be to left-anchor\n> the ref-patterns and only allow for a single wildcard character '*'\n> which matches zero or more characters (and doesn't care about slashes\n> '/'). This wildcard character should only be supported at the end of\n> the ref pattern. This means that if a client wants 'master' then they\n> would need to specify 'refs/heads/master' (and the other\n> ref_rev_parse_rules expansions) as a ref pattern. But they could say\n> \"refs/heads/*\" for all refs under refs/heads.\n\nHeh, I just responded without having read this and came up with the same\nsuggestion.\n\nSo I agree that is the right path. Or the simplification I mentioned\nthat \"refs/heads/master\" would return that ref or possibly\n\"refs/heads/master/foo\" if it exists. Remember that it's fine to be\noverly broad here. This is purely an optimization in the advertisement,\nas we'd still pick out the refs we care about in a separate step.\n\n-Peff\n"},{"id":"340343","messageId":"20180226222359.GB60857@google.com","threadId":"47524","inReplyTo":"20180223165407.2d2f4343b2497172523f9f66@google.com","subject":"Re: [PATCH v3 21/35] fetch-pack: perform a fetch using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-26T22:23:59Z","receivedAt":"2018-02-26T22:24:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/23, Jonathan Tan wrote:\n> On Tue, 6 Feb 2018 17:12:58 -0800\n> Brandon Williams wrote:\n> \n> > +\twhile ((oid = get_rev())) {\n> > +\t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n> > +\t\tif (++haves_added >= INITIAL_FLUSH)\n> > +\t\t\tbreak;\n> > +\t};\n> \n> Unnecessary semicolon after closing brace.\n\nThanks, I'll remove that.\n\n> \n> > +\t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> > +\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n> > +\t\t\t\tstate = FETCH_DONE;\n> > +\t\t\telse\n> > +\t\t\t\tstate = FETCH_SEND_REQUEST;\n> > +\t\t\tbreak;\n> \n> I haven't looked at this patch in detail, but I found a bug that can be\n> reproduced if you patch the following onto this patch:\n> \n> --- a/t/t5702-protocol-v2.sh\n> +++ b/t/t5702-protocol-v2.sh\n> @@ -124,6 +124,7 @@ test_expect_success 'clone with file:// using protocol v2' '\n> \n> test_expect_success 'fetch with file:// using protocol v2' '\n> test_commit -C file_parent two &&\n> + git -C file_parent tag -d one &&\n> \n> GIT_TRACE_PACKET=1 git -C file_child -c protocol.version=2 \\\n> fetch origin 2>log &&\n> @@ -133,7 +134,8 @@ test_expect_success 'fetch with file:// using protocol v2' '\n> test_cmp expect actual &&\n> \n> # Server responded using protocol v2\n> - grep \"fetch< version 2\" log\n> + grep \"fetch< version 2\" log &&\n> + grep \"have \" log\n> '\n> \n> Merely including the second hunk (the one with 'grep \"have \"') does not\n> make the test fail, but including both the first and second hunks does.\n> That is, fetch v2 emits \"have\" only for remote refs that point to\n> objects we already have, not for local refs.\n> \n> Everything still appears to work, except that packfiles are usually much\n> larger than they need to be.\n> \n> I did some digging in the code and found out that the equivalent of\n> find_common() (which calls `for_each_ref(rev_list_insert_ref_oid,\n> NULL)`) was not called in v2. In v1, find_common() is called immediately\n> after everything_local(), but there is no equivalent in v2. (I quoted\n> the invocation of everything_local() in v2 above.)\n\nI actually caught this Friday morning when I realized that fetching from\na referenced repository would replicated objects instead of using them\nfrom the referenced repository. Thanks for pointing this out :)\n\n-- \nBrandon Williams\n"},{"id":"340344","messageId":"xmqqk1uzqt3d.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180224040149.GA16743@sigill.intra.peff.net","subject":"Re: [PATCH v3 13/35] ls-refs: introduce ls-refs server command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-26T22:33:26Z","receivedAt":"2018-02-26T22:33:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I actually wonder if we should just specify that the patterns must\n> _always_ be fully-qualified, but may end with a single \"/*\" to iterate\n> over wildcards. Or even simpler, that \"refs/heads/foo\" would find that\n> ref itself, and anything under it.\n>\n> That drops any question about how wildcards work (e.g., does \"refs/foo*\"\n> work to find \"refs/foobar\"?).\n\nSounds quite sensible to me.\n\n"},{"id":"340351","messageId":"87inaje1uv.fsf@evledraar.gmail.com","threadId":"47524","inReplyTo":"20180224040149.GA16743@sigill.intra.peff.net","subject":"Re: [PATCH v3 13/35] ls-refs: introduce ls-refs server command","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-02-27T00:02:32Z","receivedAt":"2018-02-27T00:02:40Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Feb 24 2018, Jeff King jotted:\n\n> On Thu, Feb 22, 2018 at 04:45:14PM -0800, Brandon Williams wrote:\n>> > Does the client have to be aware that we're using wildmatch? I think\n>> > they'd need \"refs/heads/**\" to actually implement what we usually\n>> > specify in refspecs as \"refs/heads/*\". Or does the lack of WM_PATHNAME\n>> > make this work with just \"*\"?\n>> >\n>> > Do we anticipate that the client would left-anchor the refspec like\n>> > \"/refs/heads/*\" so that in theory the server could avoid looking outside\n>> > of /refs/heads/?\n>>\n>> Yeah we may want to anchor it by providing the leading '/' instead of\n>> just \"refs/\".\n>\n> I actually wonder if we should just specify that the patterns must\n> _always_ be fully-qualified, but may end with a single \"/*\" to iterate\n> over wildcards. Or even simpler, that \"refs/heads/foo\" would find that\n> ref itself, and anything under it.\n\nI agree that this is a very good trade-off for now, but I think having\nan escape hatch makes sense. It looks like the protocol is implicitly\nextendible since another parameter could be added, but maybe having such\na parameter from the get-go would make sense:\n\n pattern-type [simple|wildmatch|pcre|...]\n ref-pattern \n\nE.g.:\n\n pattern-type simple\n ref-pattern refs/tags/*\n ref-pattern refs/pull/*\n pattern-type wildmatch\n ref-pattern refs/**/2018\n pattern-type pcre\n ref-pattern ^refs/release/v-201[56789]-\\d+$\n\nI.e. each ref-pattern is typed by the pattern-type in play, with just\n\"simple\" (with the behavior being discussed here) for now, anything else\n(wildmatch, pcre etc.) would be an error.\n\nBut it allows for adding more patterns down the line, and in\ne.g. in-house setups of git where you control both the server & clients\nto make the trade-off that we'd like a bit more work on the server\n(e.g. to match dated tags created in the last 3 months) by setting some\nconfig option.\n\nThe discussion upthread about:\n\n> The other problem with tail-matching is that it's inefficient on the\n> server[...]\n\nIs also something that's only true in the current implementation, but\ndoesn't need to be, so it would be unfortunate to not work in an escape\nhatch for that limtiation.\n\nE.g. if the refs were stored indexed using the method described at\nhttps://swtch.com/~rsc/regexp/regexp4.html tail matching becomes no less\nefficient than prefix matching, but a function of how many trigrams in\nyour index match the pattern given.\n"},{"id":"340374","messageId":"20180227051555.GA65699@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"87inaje1uv.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 13/35] ls-refs: introduce ls-refs server command","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T05:15:55Z","receivedAt":"2018-02-27T05:16:55Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> On Sat, Feb 24 2018, Jeff King jotted:\n\n>> I actually wonder if we should just specify that the patterns must\n>> _always_ be fully-qualified, but may end with a single \"/*\" to iterate\n>> over wildcards. Or even simpler, that \"refs/heads/foo\" would find that\n>> ref itself, and anything under it.\n>\n> I agree that this is a very good trade-off for now, but I think having\n> an escape hatch makes sense. It looks like the protocol is implicitly\n> extendible since another parameter could be added, but maybe having such\n> a parameter from the get-go would make sense:\n\nI prefer to rely on the implicit extensibility (following the general\nYAGNI principle).\n\nIn other words, we can introduce a pattern-type later and make the\ncurrent pattern-type the default.\n\nThanks for looking to the future.\n\n[...]\n> E.g. if the refs were stored indexed using the method described at\n> https://swtch.com/~rsc/regexp/regexp4.html tail matching becomes no less\n> efficient than prefix matching, but a function of how many trigrams in\n> your index match the pattern given.\n\nI think the nearest planned change to ref storage is [1], which is\nstill optimized for prefix matching. Longer term, maybe some day\nwe'll want a secondary index that supports infix matching, or maybe\nwe'll never need it. :)\n\nSincerely,\nJonathan\n\n[1] https://public-inbox.org/git/CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com/#t\n"},{"id":"340377","messageId":"20180227054638.GB65699@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"CACsJy8BHh+Kv=om+rc6=TM60E1j9=wq4rRBvKB8HLfNTMa51Rg@mail.gmail.com","subject":"Re: [PATCH v2 12/27] serve: introduce git-serve","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T05:46:38Z","receivedAt":"2018-02-27T05:46:48Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Duy,\n\nDuy Nguyen wrote:\n> On Fri, Jan 26, 2018 at 6:58 AM, Brandon Williams wrote:\n\n>> + stateless-rpc\n>> +---------------\n>> +\n>> +If advertised, the `stateless-rpc` capability indicates that the server\n>> +supports running commands in a stateless-rpc mode, which means that a\n>> +command lasts for only a single request-response round.\n>> +\n>> +Normally a command can last for as many rounds as are required to\n>> +complete it (multiple for negotiation during fetch or no additional\n>> +trips in the case of ls-refs). If the client sends the `stateless-rpc`\n>> +capability with a value of `true` (in the form `stateless-rpc=true`)\n>> +then the invoked command must only last a single round.\n>\n> Speaking of stateless-rpc, I remember last time this topic was brought\n> up, there was some discussion to kind of optimize it for http as well,\n> to fit the \"client sends request, server responds data\" model and\n> avoid too many round trips (ideally everything happens in one round\n> trip). Does it evolve to anything real? All the cool stuff happened\n> while I was away, sorry if this was discussed and settled.\n\nWe have a few different ideas for improving negotiation. They were\nspeculative enough that we didn't want to make them part of the\nbaseline protocol v2. Feel free to poke me in a new thread. :)\n\nSome teasers:\n\n- allow both client and server to suggest commits in negotiation,\n instead of just the client?\n\n- send a bloom filter for the peer to filter their suggestions\n against?\n\n- send other basic information like maximum generation number or\n maximum commit date?\n\n- exponential backoff in negotiation instead of linear walking?\n prioritizing ref tips? Imitating the bitmap selection algorithm?\n\n- at the \"end\" of negotiation, sending a graph data structure instead\n of a pack, to allow an extra round trip to produce a truly minimal\n pack?\n\nThose are some initial ideas, but it's also likely that someone can\ncome up with some other experiments to try, too. (E.g. we've looked\nat various papers on set reconciliation, but they don't make enough\nuse of the graph structure to help much.)\n\nThanks,\nJonathan\n"},{"id":"340379","messageId":"20180227055746.GC65699@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-3-bmwill@google.com","subject":"Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T05:57:46Z","receivedAt":"2018-02-27T05:57:58Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> Sometimes it is advantageous to be able to peek the next packet line\n> without consuming it (e.g. to be able to determine the protocol version\n> a server is speaking). In order to do that introduce 'struct\n> packet_reader' which is an abstraction around the normal packet reading\n> logic. This enables a caller to be able to peek a single line at a time\n> using 'packet_reader_peek()' and having a caller consume a line by\n> calling 'packet_reader_read()'.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> pkt-line.h | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 117 insertions(+)\n\nI like it!\n\nThe questions and nits from\nhttps://public-inbox.org/git/20180213004937.GB42272@aiede.svl.corp.google.com/\nstill apply. In particular, the ownership of the buffers inside the\n'struct packet_reader' is still unclear; could the packet_reader create\nits own (strbuf) buffers so that the contract around them (who is allowed\nto write to them; who is responsible for freeing them) is more obvious?\n\nThanks,\nJonathan\n"},{"id":"340380","messageId":"20180227060840.GD65699@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-7-bmwill@google.com","subject":"Re: [PATCH v3 06/35] transport: use get_refs_via_connect to get refs","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T06:08:40Z","receivedAt":"2018-02-27T06:08:49Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Remove code duplication and use the existing 'get_refs_via_connect()'\n> function to retrieve a remote's heads in 'fetch_refs_via_pack()' and\n> 'git_transport_push()'.\n> \n> Signed-off-by: Brandon Williams \n> ---\n> transport.c | 18 ++++--------------\n> 1 file changed, 4 insertions(+), 14 deletions(-)\n\nI like the diffstat.\n\n[...]\n> +++ b/transport.c\n> @@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n> \targs.cloning = transport->cloning;\n> \targs.update_shallow = data->options.update_shallow;\n> \n> -\tif (!data->got_remote_heads) {\n> -\t\tconnect_setup(transport, 0);\n> -\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n> -\t\t\t\t NULL, &data->shallow);\n> -\t\tdata->got_remote_heads = 1;\n> -\t}\n> +\tif (!data->got_remote_heads)\n> +\t\trefs_tmp = get_refs_via_connect(transport, 0);\n\nThe only difference between the old and new code is that the old code\npasses NULL as 'extra_have' and the new code passes &data->extra_have.\n\nThat means this populates the data->extra_have oid_array. Does it\nmatter?\n\n> @@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n> \tstruct send_pack_args args;\n> \tint ret;\n> \n> -\tif (!data->got_remote_heads) {\n> -\t\tstruct ref *tmp_refs;\n> -\t\tconnect_setup(transport, 1);\n> -\n> -\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n> -\t\t\t\t NULL, &data->shallow);\n> -\t\tdata->got_remote_heads = 1;\n> -\t}\n> +\tif (!data->got_remote_heads)\n> +\t\tget_refs_via_connect(transport, 1);\n\nnot a new problem, just curious: Does this leak tmp_refs?\n\nSame question as the other caller about whether we mind getting\nextra_have populated.\n\nThanks,\nJonathan\n"},{"id":"340381","messageId":"20180227061254.GE65699@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180227055746.GC65699@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T06:12:54Z","receivedAt":"2018-02-27T06:13:02Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n> Brandon Williams wrote:\n\n>> Sometimes it is advantageous to be able to peek the next packet line\n>> without consuming it (e.g. to be able to determine the protocol version\n>> a server is speaking). In order to do that introduce 'struct\n>> packet_reader' which is an abstraction around the normal packet reading\n>> logic. This enables a caller to be able to peek a single line at a time\n>> using 'packet_reader_peek()' and having a caller consume a line by\n>> calling 'packet_reader_read()'.\n>>\n>> Signed-off-by: Brandon Williams \n>> ---\n>> pkt-line.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n>> pkt-line.h | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n>> 2 files changed, 117 insertions(+)\n>\n> I like it!\n>\n> The questions and nits from\n> https://public-inbox.org/git/20180213004937.GB42272@aiede.svl.corp.google.com/\n> still apply. In particular, the ownership of the buffers inside the\n> 'struct packet_reader' is still unclear; could the packet_reader create\n> its own (strbuf) buffers so that the contract around them (who is allowed\n> to write to them; who is responsible for freeing them) is more obvious?\n\nJust to be clear: I sent that review after you sent this patch, so\nthere should not have been any reason for me to expect the q's and\nnits to magically not apply. ;-)\n"},{"id":"340382","messageId":"20180227061813.GF65699@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-11-bmwill@google.com","subject":"Re: [PATCH v3 10/35] protocol: introduce enum protocol_version value protocol_v2","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T06:18:13Z","receivedAt":"2018-02-27T06:18:22Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> Introduce protocol_v2, a new value for 'enum protocol_version'.\n> Subsequent patches will fill in the implementation of protocol_v2.\n>\n> Signed-off-by: Brandon Williams \n> ---\n\nYay!\n\n[...]\n> +++ b/builtin/fetch-pack.c\n> @@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> \t\t\t PACKET_READ_GENTLE_ON_EOF);\n> \n> \tswitch (discover_version(&reader)) {\n> +\tcase protocol_v2:\n> +\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tbreak;\n\nThis code goes away in a later patch, so no need to do anything about\nthis, but the 'break' is redundant after the 'die'.\n\n[...]\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> \t\tunpack_limit = receive_unpack_limit;\n> \n> \tswitch (determine_protocol_version_server()) {\n> +\tcase protocol_v2:\n> +\t\t/*\n> +\t\t * push support for protocol v2 has not been implemented yet,\n> +\t\t * so ignore the request to use v2 and fallback to using v0.\n> +\t\t */\n> +\t\tbreak;\n\nAs you mentioned in the cover letter, it's probably worth doing the\nsame fallback on the client side (send-pack), too.\n\nOtherwise when this client talks to a new-enough server, it would\nrequest protocol v2 and then get confused when the server responds\nwith the protocol v2 it requested.\n\nThanks,\nJonathan\n"},{"id":"340383","messageId":"20180227062128.GG65699@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180222112533.0d7c6023fb8e4098efedfe31@google.com","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T06:21:28Z","receivedAt":"2018-02-27T06:21:36Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Tan wrote:\n> On Thu, 22 Feb 2018 13:26:58 -0500\n> Jeff King wrote:\n\n>> I agree that it shouldn't matter much here. But if the name argv_array\n>> is standing in the way of using it, I think we should consider giving it\n>> a more general name. I picked that not to evoke \"this must be arguments\"\n>> but \"this is terminated by a single NULL\".\n[...]\n> This sounds reasonable - I withdraw my comment about using struct\n> string_list.\n\nMarking with #leftoverbits as a reminder to think about what such a\nmore general name would be (or what kind of docs to put in\nargv-array.h) and make it so the next time I do a search for that\nkeyword.\n\nThanks,\nJonathan\n"},{"id":"340384","messageId":"20180227065158.GH65699@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-15-bmwill@google.com","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T06:51:58Z","receivedAt":"2018-02-27T06:52:07Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Teach the client to be able to request a remote's refs using protocol\n> v2. This is done by having a client issue a 'ls-refs' request to a v2\n> server.\n\nYay, ls-remote support!\n\n[...]\n> --- a/builtin/upload-pack.c\n> +++ b/builtin/upload-pack.c\n> @@ -5,6 +5,7 @@\n> #include \"parse-options.h\"\n> #include \"protocol.h\"\n> #include \"upload-pack.h\"\n> +#include \"serve.h\"\n\nnit, no change needed in this patch: What is a good logical order for\nthe #includes here? Bonus points if there's a tool to make it happen\nautomatically.\n\nAsking since adding #includes like this at the end tends to result in\na harder-to-read list of #includes, sometimes with duplicates, and\noften producing conflicts when multiple patches in flight add a\n#include to the same file.\n\n[...]\n> @@ -48,11 +50,9 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n> \n> \tswitch (determine_protocol_version_server()) {\n> \tcase protocol_v2:\n> -\t\t/*\n> -\t\t * fetch support for protocol v2 has not been implemented yet,\n> -\t\t * so ignore the request to use v2 and fallback to using v0.\n> -\t\t */\n> -\t\tupload_pack(&opts);\n> +\t\tserve_opts.advertise_capabilities = opts.advertise_refs;\n> +\t\tserve_opts.stateless_rpc = opts.stateless_rpc;\n> +\t\tserve(&serve_opts);\n\nInteresting! daemon.c has its own (file-local) serve() function;\ncan one of the two be renamed?\n\nI actually like both names: serve() is a traditional name for the\nfunction a server calls when it's done setting up and is ready to\nserve. But the clash is confusing.\n\n[...]\n> +++ b/connect.c\n> @@ -12,9 +12,11 @@\n> #include \"sha1-array.h\"\n> #include \"transport.h\"\n> #include \"strbuf.h\"\n> +#include \"version.h\"\n> #include \"protocol.h\"\n> \n> static char *server_capabilities;\n> +static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n\nCan a quick doc comment describe these and how they relate?\n\nIs only one of them set, based on which protocol version is in use?\nShould server_capabilities be renamed to server_capabilities_v1?\n\n> static const char *parse_feature_value(const char *, const char *, int *);\n> \n> static int check_ref(const char *name, unsigned int flags)\n> @@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)\n> \t\t \"and the repository exists.\"));\n> }\n> \n> +/* Checks if the server supports the capability 'c' */\n> +int server_supports_v2(const char *c, int die_on_error)\n> +{\n> +\tint i;\n> +\n> +\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n> +\t\tconst char *out;\n> +\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n> +\t\t (!*out || *out == '='))\n> +\t\t\treturn 1;\n> +\t}\n> +\n> +\tif (die_on_error)\n> +\t\tdie(\"server doesn't support '%s'\", c);\n> +\n> +\treturn 0;\n> +}\n\nNice.\n> +\n> +static void process_capabilities_v2(struct packet_reader *reader)\n> +{\n> +\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL)\n> +\t\targv_array_push(&server_capabilities_v2, reader->line);\n> +\n> +\tif (reader->status != PACKET_READ_FLUSH)\n> +\t\tdie(\"protocol error\");\n\nCan this say more? E.g. \"expected flush after capabilities, got \"?\n\n[...]\n> @@ -85,7 +114,7 @@ enum protocol_version discover_version(struct packet_reader *reader)\n> \t/* Maybe process capabilities here, at least for v2 */\n\nIs this comment out of date now?\n\n> \tswitch (version) {\n> \tcase protocol_v2:\n> -\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\tprocess_capabilities_v2(reader);\n> \t\tbreak;\n> \tcase protocol_v1:\n> \t\t/* Read the peeked version line */\n> @@ -293,6 +322,98 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n> \treturn list;\n> }\n> \n> +static int process_ref_v2(const char *line, struct ref ***list)\n\nWhat does the return value represent?\n\nCould it return the more typical 0 on success, -1 on error?\n\n> +{\n> +\tint ret = 1;\n> +\tint i = 0;\n> +\tstruct object_id old_oid;\n> +\tstruct ref *ref;\n> +\tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n> +\n> +\tif (string_list_split(&line_sections, line, ' ', -1) < 2) {\n\nCan there be a comment describing the expected format?\n\n> +\t\tret = 0;\n> +\t\tgoto out;\n> +\t}\n> +\n> +\tif (get_oid_hex(line_sections.items[i++].string, &old_oid)) {\n> +\t\tret = 0;\n> +\t\tgoto out;\n> +\t}\n> +\n> +\tref = alloc_ref(line_sections.items[i++].string);\n\nRef names cannot contains a space, so this is safe. Good.\n\n> +\n> +\toidcpy(&ref->old_oid, &old_oid);\n> +\t**list = ref;\n> +\t*list = &ref->next;\n> +\n> +\tfor (; i < line_sections.nr; i++) {\n> +\t\tconst char *arg = line_sections.items[i].string;\n> +\t\tif (skip_prefix(arg, \"symref-target:\", &arg))\n> +\t\t\tref->symref = xstrdup(arg);\n\nUsing space-delimited fields in a single pkt-line means that\n- values cannot contains a space\n- total length is limited by the size of a pkt-line\n\nGiven the context, I think that's fine. More generally it is tempting\nto use a pkt-line per field to avoid the trouble v1 had with\ncapability lists crammed into a pkt-line, but I see why you used a\npkt-line per ref to avoid having to have sections-within-a-section.\n\nMy only potential worry is the length part: do we have an explicit\nlimit on the length of a ref name? git-check-ref-format(1) doesn't\nmention one. A 32k ref name would be a bit ridiculous, though.\n\n> +\n> +\t\tif (skip_prefix(arg, \"peeled:\", &arg)) {\n> +\t\t\tstruct object_id peeled_oid;\n> +\t\t\tchar *peeled_name;\n> +\t\t\tstruct ref *peeled;\n> +\t\t\tif (get_oid_hex(arg, &peeled_oid)) {\n> +\t\t\t\tret = 0;\n> +\t\t\t\tgoto out;\n> +\t\t\t}\n\nCan this also check that there's no trailing garbage after the oid?\n\n[...]\n> +\n> +\t\t\tpeeled_name = xstrfmt(\"%s^{}\", ref->name);\n\noptional: can reuse a buffer to avoid allocation churn:\n\n\tstruct strbuf peeled_name = STRBUF_INIT;\n\n\t\t\tstrbuf_reset(&peeled_name);\n\t\t\tstrbuf_addf(&peeled_name, \"%s^{}\", ref->name);\n\t\t\t// or strbuf_addstr(ref->name); strbuf_addstr(\"^{}\");\n\n out:\n \tstrbuf_release(&peeled_name);\n\n[...]\n> +struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> +\t\t\t struct ref **list, int for_push,\n> +\t\t\t const struct argv_array *ref_patterns)\n> +{\n> +\tint i;\n> +\t*list = NULL;\n> +\n> +\t/* Check that the server supports the ls-refs command */\n> +\t/* Issue request for ls-refs */\n> +\tif (server_supports_v2(\"ls-refs\", 1))\n> +\t\tpacket_write_fmt(fd_out, \"command=ls-refs\\n\");\n\nSince the code is so clear, I don't think the above two comments are\nhelping.\n\n> +\n> +\tif (server_supports_v2(\"agent\", 0))\n> +\t packet_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n\nwhitespace nit: mixing tabs and spaces. Does \"make style\" catch this?\n\n> +\n> +\tpacket_delim(fd_out);\n> +\t/* When pushing we don't want to request the peeled tags */\n\nCan you say more about this? In theory it would be nice to have the\npeeled tags since they name commits whose history can be excluded from\nthe pack.\n\n> +\tif (!for_push)\n> +\t\tpacket_write_fmt(fd_out, \"peel\\n\");\n> +\tpacket_write_fmt(fd_out, \"symrefs\\n\");\n\nAre symrefs useful during push?\n\n> +\tfor (i = 0; ref_patterns && i < ref_patterns->argc; i++) {\n> +\t\tpacket_write_fmt(fd_out, \"ref-pattern %s\\n\",\n> +\t\t\t\t ref_patterns->argv[i]);\n> +\t}\n\nThe exciting part.\n\nWhy do these pkts end with \\n? I would have expected the pkt-line\nframing to work to delimit them.\n\n> +\tpacket_flush(fd_out);\n> +\n> +\t/* Process response from server */\n> +\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n> +\t\tif (!process_ref_v2(reader->line, &list))\n> +\t\t\tdie(\"invalid ls-refs response: %s\", reader->line);\n> +\t}\n> +\n> +\tif (reader->status != PACKET_READ_FLUSH)\n> +\t\tdie(\"protocol error\");\n\nCan this protocol error give more detail? When diagnosing an error in\nservers, proxies, or lower-level networking issues, informative protocol\nerrors can be very helpful (similar to syntax errors from a compiler).\n\n[...]\n> --- a/connect.h\n> +++ b/connect.h\n> @@ -16,4 +16,6 @@ extern int url_is_local_not_ssh(const char *url);\n> struct packet_reader;\n> extern enum protocol_version discover_version(struct packet_reader *reader);\n> \n> +extern int server_supports_v2(const char *c, int die_on_error);\n\nconst char *cap, maybe?\n\n[...]\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -151,10 +151,14 @@ void free_refs(struct ref *ref);\n> \n> struct oid_array;\n> struct packet_reader;\n> +struct argv_array;\n> extern struct ref **get_remote_heads(struct packet_reader *reader,\n> \t\t\t\t struct ref **list, unsigned int flags,\n> \t\t\t\t struct oid_array *extra_have,\n> \t\t\t\t struct oid_array *shallow_points);\n> +extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> +\t\t\t\t struct ref **list, int for_push,\n> +\t\t\t\t const struct argv_array *ref_patterns);\n\nWhat is the difference between get_remote_heads and get_remote_refs?\nA comment might help. (BTW, thanks for making the new saner name to\nreplace get_remote_heads!)\n\n[...]\n> --- /dev/null\n> +++ b/t/t5702-protocol-v2.sh\n> @@ -0,0 +1,53 @@\n> +#!/bin/sh\n> +\n> +test_description='test git wire-protocol version 2'\n\nWoot!\n\n[...]\n> +test_expect_success 'list refs with git:// using protocol v2' '\n> +\tGIT_TRACE_PACKET=1 git -c protocol.version=2 \\\n> +\t\tls-remote --symref \"$GIT_DAEMON_URL/parent\" >actual 2>log &&\n> +\n> +\t# Client requested to use protocol v2\n> +\tgrep \"git> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n> +\t# Server responded using protocol v2\n> +\tgrep \"git< version 2\" log &&\n\noptional: Could anchor these greps to make the test tighter (e.g. to\nnot match \"version 20\".\n\nThanks,\nJonathan\n"},{"id":"340385","messageId":"20180227065342.GI65699@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-19-bmwill@google.com","subject":"Re: [PATCH v3 18/35] fetch: pass ref patterns when fetching","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T06:53:42Z","receivedAt":"2018-02-27T06:53:51Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Construct a list of ref patterns to be passed to\n> 'transport_get_remote_refs()' from the refspec to be used during the\n> fetch. This list of ref patterns will be used to allow the server to\n> filter the ref advertisement when communicating using protocol v2.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> builtin/fetch.c | 12 +++++++++++-\n> 1 file changed, 11 insertions(+), 1 deletion(-)\n\nReviewed-by: Jonathan Nieder \nNice.\n\nI take it that tests covering this come later in the series?\n"},{"id":"340413","messageId":"20180227180216.GA209668@google.com","threadId":"47524","inReplyTo":"20180227051555.GA65699@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 13/35] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-27T18:02:16Z","receivedAt":"2018-02-27T18:02:26Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/26, Jonathan Nieder wrote:\n> Ævar Arnfjörð Bjarmason wrote:\n> > On Sat, Feb 24 2018, Jeff King jotted:\n> \n> >> I actually wonder if we should just specify that the patterns must\n> >> _always_ be fully-qualified, but may end with a single \"/*\" to iterate\n> >> over wildcards. Or even simpler, that \"refs/heads/foo\" would find that\n> >> ref itself, and anything under it.\n> >\n> > I agree that this is a very good trade-off for now, but I think having\n> > an escape hatch makes sense. It looks like the protocol is implicitly\n> > extendible since another parameter could be added, but maybe having such\n> > a parameter from the get-go would make sense:\n> \n> I prefer to rely on the implicit extensibility (following the general\n> YAGNI principle).\n> \n> In other words, we can introduce a pattern-type later and make the\n> current pattern-type the default.\n\nYeah this is what I'm going to do for the next re-roll of the series,\nmake the pattern matching simple and later we can extend it if we want\nsince we already have the ability to add new features to commands (you\ncan see how I added shallow to fetch for an example).\n\n> \n> Thanks for looking to the future.\n> \n> [...]\n> > E.g. if the refs were stored indexed using the method described at\n> > https://swtch.com/~rsc/regexp/regexp4.html tail matching becomes no less\n> > efficient than prefix matching, but a function of how many trigrams in\n> > your index match the pattern given.\n> \n> I think the nearest planned change to ref storage is [1], which is\n> still optimized for prefix matching. Longer term, maybe some day\n> we'll want a secondary index that supports infix matching, or maybe\n> we'll never need it. :)\n> \n> Sincerely,\n> Jonathan\n> \n> [1] https://public-inbox.org/git/CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com/#t\n\n-- \nBrandon Williams\n"},{"id":"340414","messageId":"20180227100555.91f2a5e196bff3968871a16e@google.com","threadId":"47524","inReplyTo":"20180223213315.GE234838@google.com","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-27T18:05:55Z","receivedAt":"2018-02-27T18:06:03Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 23 Feb 2018 13:33:15 -0800\nBrandon Williams wrote:\n\n> On 02/21, Jonathan Tan wrote:\n> > As someone who is implementing the server side of protocol V2 in JGit, I\n> > now have a bit more insight into this :-)\n> > \n> > First of all, I used to not have a strong opinion on the existence of a\n> > new endpoint, but now I think that it's better to *not* have git-serve.\n> > As it is, as far as I can tell, upload-pack also needs to support (and\n> > does support, as of the end of this patch set) protocol v2 anyway, so it\n> > might be better to merely upgrade upload-pack.\n> \n> Having it allows for easier testing and the easy ability to make it a\n> true endpoint when we want to. As of right now, git-serve isn't an\n> endpoint as you can't issue requests there via http-backend or\n> git-daemon.\n\nIs git-serve planned to be a new endpoint?\n\nIf yes, I now don't think it's a good idea - it's an extra burden to\nreimplementors without much benefit (to have a new endpoint that does\nthe same things as upload-pack).\n\nIf not, I don't think that easier testing makes it worth having an extra\nbinary. Couldn't the same tests be done by running upload-pack directly?\n"},{"id":"340417","messageId":"20180227181428.GB209668@google.com","threadId":"47524","inReplyTo":"20180213004937.GB42272@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-27T18:14:28Z","receivedAt":"2018-02-27T18:14:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/12, Jonathan Nieder wrote:\n> [...]\n> > --- a/pkt-line.h\n> > +++ b/pkt-line.h\n> > @@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n> > */\n> > ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);\n> > \n> > +struct packet_reader {\n> > +\t/* source file descriptor */\n> > +\tint fd;\n> > +\n> > +\t/* source buffer and its size */\n> > +\tchar *src_buffer;\n> > +\tsize_t src_len;\n> \n> Can or should this be a strbuf?\n> \n> > +\n> > +\t/* buffer that pkt-lines are read into and its size */\n> > +\tchar *buffer;\n> > +\tunsigned buffer_size;\n> \n> Likewise.\n> \n\nThis struct is setup to be a drop in replacement for the existing\nread_packet() family of functions. Because of this I tried to make the\ninterface as similar as possible to make it easy to convert to using it\nas well as having no need to clean anything up (because the struct is\nreally just a wrapper and doesn't own anything).\n\n-- \nBrandon Williams\n"},{"id":"340418","messageId":"20180227181715.GC209668@google.com","threadId":"47524","inReplyTo":"20180227060840.GD65699@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 06/35] transport: use get_refs_via_connect to get refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-27T18:17:15Z","receivedAt":"2018-02-27T18:17:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/26, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> \n> > Remove code duplication and use the existing 'get_refs_via_connect()'\n> > function to retrieve a remote's heads in 'fetch_refs_via_pack()' and\n> > 'git_transport_push()'.\n> > \n> > Signed-off-by: Brandon Williams \n> > ---\n> > transport.c | 18 ++++--------------\n> > 1 file changed, 4 insertions(+), 14 deletions(-)\n> \n> I like the diffstat.\n> \n> [...]\n> > +++ b/transport.c\n> > @@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n> > \targs.cloning = transport->cloning;\n> > \targs.update_shallow = data->options.update_shallow;\n> > \n> > -\tif (!data->got_remote_heads) {\n> > -\t\tconnect_setup(transport, 0);\n> > -\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n> > -\t\t\t\t NULL, &data->shallow);\n> > -\t\tdata->got_remote_heads = 1;\n> > -\t}\n> > +\tif (!data->got_remote_heads)\n> > +\t\trefs_tmp = get_refs_via_connect(transport, 0);\n> \n> The only difference between the old and new code is that the old code\n> passes NULL as 'extra_have' and the new code passes &data->extra_have.\n> \n> That means this populates the data->extra_have oid_array. Does it\n> matter?\n> \n> > @@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n> > \tstruct send_pack_args args;\n> > \tint ret;\n> > \n> > -\tif (!data->got_remote_heads) {\n> > -\t\tstruct ref *tmp_refs;\n> > -\t\tconnect_setup(transport, 1);\n> > -\n> > -\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n> > -\t\t\t\t NULL, &data->shallow);\n> > -\t\tdata->got_remote_heads = 1;\n> > -\t}\n> > +\tif (!data->got_remote_heads)\n> > +\t\tget_refs_via_connect(transport, 1);\n> \n> not a new problem, just curious: Does this leak tmp_refs?\n\nMaybe, though its removed by this patch.\n\n> \n> Same question as the other caller about whether we mind getting\n> extra_have populated.\n\nI don't think its a problem to have extra_have populated, least I\nhaven't seen anything to lead me to believe it would be a problem.\n\n-- \nBrandon Williams\n"},{"id":"340419","messageId":"CAGZ79kaWu=M_EJH=wKzYn2YV70GO0iuS_T68nztp5fYtiVA9mg@mail.gmail.com","threadId":"47524","inReplyTo":"20180207011312.189834-20-bmwill@google.com","subject":"Re: [PATCH v3 19/35] push: pass ref patterns when pushing","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-27T18:23:02Z","receivedAt":"2018-02-27T18:23:09Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> Construct a list of ref patterns to be passed to 'get_refs_list()' from\n> the refspec to be used during the push. This list of ref patterns will\n> be used to allow the server to filter the ref advertisement when\n> communicating using protocol v2.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> transport.c | 17 ++++++++++++++++-\n> 1 file changed, 16 insertions(+), 1 deletion(-)\n>\n> diff --git a/transport.c b/transport.c\n> index dfc603b36..6ea3905e3 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -1026,11 +1026,26 @@ int transport_push(struct transport *transport,\n> int porcelain = flags & TRANSPORT_PUSH_PORCELAIN;\n> int pretend = flags & TRANSPORT_PUSH_DRY_RUN;\n> int push_ret, ret, err;\n> + struct refspec *tmp_rs;\n> + struct argv_array ref_patterns = ARGV_ARRAY_INIT;\n> + int i;\n>\n> if (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n> return -1;\n>\n> - remote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n> + tmp_rs = parse_push_refspec(refspec_nr, refspec);\n> + for (i = 0; i < refspec_nr; i++) {\n> + if (tmp_rs[i].dst)\n> + argv_array_push(&ref_patterns, tmp_rs[i].dst);\n> + else if (tmp_rs[i].src && !tmp_rs[i].exact_sha1)\n> + argv_array_push(&ref_patterns, tmp_rs[i].src);\n\nelse /* !tmp_rs[i].dst && (!tmp_rs[i].src || tmp_rs[i].exact_sha1)\n\nI would think the case of !dst && !src cannot happen, as then there is\nno refspec, but what about the !!exact_sha1 case ?\n\nI'd think that is something like\n\n git push origin $(git rev-parse HEAD)\n\nfor which I'd think we'd bail out anyway?\nBut that would happen at a different place, here we can ignore\nthe exact hashes for listing refs purposes.\n\nCan you add a comment or rather explain in the commit\nmessage to make this less confusing?\n\nStefan\n"},{"id":"340420","messageId":"20180227182854.GA174036@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-23-bmwill@google.com","subject":"Re: [PATCH v3 22/35] upload-pack: support shallow requests","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T18:29:49Z","receivedAt":"2018-02-27T18:30:00Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> Add the 'shallow' feature to the protocol version 2 command 'fetch'\n> which indicates that the server supports shallow clients and deepen\n> requets.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> Documentation/technical/protocol-v2.txt | 67 +++++++++++++++-\n> serve.c | 2 +-\n> t/t5701-git-serve.sh | 2 +-\n> upload-pack.c | 138 +++++++++++++++++++++++---------\n> upload-pack.h | 3 +\n> 5 files changed, 173 insertions(+), 39 deletions(-)\n\nYay! We've been running with this for a while at Google (for file://\nfetches, at least) and it's been working well.\n\n[...]\n> --- a/Documentation/technical/protocol-v2.txt\n> +++ b/Documentation/technical/protocol-v2.txt\n> @@ -201,12 +201,42 @@ packet-lines:\n> \tto its base by position in pack rather than by an oid. That is,\n> \tthey can read OBJ_OFS_DELTA (ake type 6) in a packfile.\n> \n> + shallow \n> +\tA client must notify the server of all objects for which it only\n> +\thas shallow copies of (meaning that it doesn't have the parents\n\nGrammar nit: \"for which it only has shallow copies of\" should be e.g.\n\"for which it only has shallow copies\" or \"that it only has shallow\ncopies of\" or \"that it only has shallow copies for\".\n\nI think s/objects/commits/ would also help clarify.\n\n> +\tof a commit) by supplying a 'shallow ' line for each such\n> +\tobject so that the serve is aware of the limitations of the\n\ns/serve/server/\n\n> +\tclient's history.\n\nIs it worth mentioning that this is about negotiation? E.g. \"so that\nthe server is aware that the client may not have all objects reachable\nfrom such commits\".\n\n> +\n> + deepen \n> +\tRequest that the fetch/clone should be shallow having a commit depth of\n\nnit: s/Request/Requests/, for consistency with the others?\n\n> +\t relative to the remote side.\n\nWhat does the value of mean? E.g. does a depth of 1 mean to\nfetch only the commits named in \"have\", 2 to fetch those commits plus\ntheir parents, etc, or am I off by one?\n\nIs always a positive number?\n\nWhat happens if starts with a 0? Is that a client error?\n\n> +\n> + deepen-relative\n> +\tRequests that the semantics of the \"deepen\" command be changed\n> +\tto indicate that the depth requested is relative to the clients\n> +\tcurrent shallow boundary, instead of relative to the remote\n> +\trefs.\n\ns/clients/client's/\n\ns/remote refs/requested commits/ or \"wants\" or something.\n\n> +\n> + deepen-since \n> +\tRequests that the shallow clone/fetch should be cut at a\n> +\tspecific time, instead of depth. Internally it's equivalent of\n> +\tdoing \"rev-list --max-age=\". Cannot be used with\n> +\t\"deepen\".\n\nNits:\n s/rev-list/git rev-list/\n s/equivalent of/equivalent to/ or 'the equivalent of'.\n\nSince the git-rev-list(1) manpage doesn't tell me: what is the format\nof ? And is the requested time interval inclusive of\nexclusive?\n\n> +\n> + deepen-not \n> +\tRequests that the shallow clone/fetch should be cut at a\n> +\tspecific revision specified by '', instead of a depth.\n> +\tInternally it's equivalent of doing \"rev-list --not \".\n> +\tCannot be used with \"deepen\", but can be used with\n> +\t\"deepen-since\".\n\nInteresting.\n\nnit: s/rev-list/git rev-list/\n\nWhat is the format of ? E.g. can it be an arbitrary revision\nspecifier or is it an oid?\n\n[...]\n> output = *section\n> - section = (acknowledgments | packfile)\n> + section = (acknowledgments | shallow-info | packfile)\n> \t (flush-pkt | delim-pkt)\n\nIt looks like sections can go in an arbitrary order. Are there\ntests to make sure the server can cope with reordering? (I ask\nnot because I mistrust the server but because I have some vague\nhope that other server implementations might be inspired by our\ntests.)\n\n[...]\n> @@ -215,6 +245,11 @@ header.\n> nak = PKT-LINE(\"NAK\" LF)\n> ack = PKT-LINE(\"ACK\" SP obj-id LF)\n> \n> + shallow-info = PKT-LINE(\"shallow-info\" LF)\n> +\t\t *PKT-LINE((shallow | unshallow) LF)\n> + shallow = \"shallow\" SP obj-id\n> + unshallow = \"unshallow\" SP obj-id\n\nLikewise: it looks like shallows and unshallows can be intermixed; can\nthis be either (a) tightened or (b) covered by tests to make sure a\nlater refactoring doesn't accidentally tighten it?\n\n[...]\n> @@ -247,6 +282,36 @@ header.\n> \t determined the objects it plans to send to the client and no\n> \t further negotiation is needed.\n> \n> +----\n> + shallow-info section\n> +\tIf the client has requested a shallow fetch/clone, a shallow\n> +\tclient requests a fetch or the server is shallow then the\n> +\tserver's response may include a shallow-info section.\n\nI'm having trouble parsing this sentence.\n\n> The\n> +\tshallow-info section will be include if (due to one of the above\n\nnit: s/include/included/\n\n> +\tconditions) the server needs to inform the client of any shallow\n> +\tboundaries or adjustments to the clients already existing\n> +\tshallow boundaries.\n> +\n> +\t* Always begins with the section header \"shallow-info\"\n> +\n> +\t* If a positive depth is requested, the server will compute the\n> +\t set of commits which are no deeper than the desired depth.\n> +\n> +\t* The server sends a \"shallow obj-id\" line for each commit whose\n> +\t parents will not be sent in the following packfile.\n> +\n> +\t* The server sends an \"unshallow obj-id\" line for each commit\n> +\t which the client has indicated is shallow, but is no longer\n> +\t shallow as a result of the fetch (due to its parents being\n> +\t sent in the following packfile).\n> +\n> +\t* The server MUST NOT send any \"unshallow\" lines for anything\n> +\t which the client has not indicated was shallow as a part of\n> +\t its request.\n> +\n> +\t* This section is only included if a packfile section is also\n> +\t included in the response.\n\nNeat.\n\nI wonder if shallow information is also useful for negotiation. I\nguess mostly not today, since the client is the only one that suggests\n\"have\"s for negotiation and although it could accidentally exclude too\nmany \"have\"s by going down a path from a server's ack, that feels like\na rare case.\n\n[...]\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -710,7 +710,6 @@ static void deepen(int depth, int deepen_relative,\n> \t}\n> \n> \tsend_unshallow(shallows);\n> -\tpacket_flush(1);\n\nWhat does this part do?\n> }\n> \n> static void deepen_by_rev_list(int ac, const char **av,\n> @@ -722,7 +721,52 @@ static void deepen_by_rev_list(int ac, const char **av,\n> \tsend_shallow(result);\n> \tfree_commit_list(result);\n> \tsend_unshallow(shallows);\n> -\tpacket_flush(1);\n\nSame question.\n\n> +}\n> +\n> +static int send_shallow_list(int depth, int deepen_rev_list,\n> +\t\t\t timestamp_t deepen_since,\n> +\t\t\t struct string_list *deepen_not,\n> +\t\t\t struct object_array *shallows)\n\nWhat does the return value from this function represent? It doesn't\nappear to be the usual \"0 means success, -1 means failure\" so a\ncomment would help.\n\n> +{\n> +\tint ret = 0;\n> +\n> +\tif (depth > 0 && deepen_rev_list)\n> +\t\tdie(\"git upload-pack: deepen and deepen-since (or deepen-not) cannot be used together\");\n\nnit: long line (can/should \"make style\" find these?)\n\nThe error message is pretty long, longer than a typical 80-column\nterminal, so probably best to find a way to make the message shorter.\nE.g.\n\n\t\tdie(\"upload-pack: deepen cannot be combined with other deepen-* options\");\n\nThat still would be >80 columns with the indent, so the usual style\nwould be to break it into multiple strings and use C preprocessor\nconcatenation (yuck):\n\n\t\tdie(\"upload-pack: \"\n\t\t \"deepen cannot be combined with other deepen-* options\");\n\n[...]\n> +\tif (depth > 0) {\n> +\t\tdeepen(depth, deepen_relative, shallows);\n> +\t\tret = 1;\n> +\t} else if (deepen_rev_list) {\n> +\t\tstruct argv_array av = ARGV_ARRAY_INIT;\n> +\t\tint i;\n> +\n> +\t\targv_array_push(&av, \"rev-list\");\n> +\t\tif (deepen_since)\n> +\t\t\targv_array_pushf(&av, \"--max-age=%\"PRItime, deepen_since);\n> +\t\tif (deepen_not->nr) {\n> +\t\t\targv_array_push(&av, \"--not\");\n> +\t\t\tfor (i = 0; i < deepen_not->nr; i++) {\n> +\t\t\t\tstruct string_list_item *s = deepen_not->items + i;\n> +\t\t\t\targv_array_push(&av, s->string);\n\nThis accepts arbitrary rev-list arguments, which feels dangerous\n(could end up doing an expensive operation or reading arbitrary files\nor finding a way to execute arbitrary code).\n\n[...]\n> -\t\tif (deepen_not.nr) {\n> -\t\t\targv_array_push(&av, \"--not\");\n> -\t\t\tfor (i = 0; i < deepen_not.nr; i++) {\n> -\t\t\t\tstruct string_list_item *s = deepen_not.items + i;\n> -\t\t\t\targv_array_push(&av, s->string);\n\nHuh. Looks like some of the above comments are better addressed to an\nearlier patch.\n\n[...]\n> @@ -1071,6 +1085,13 @@ struct upload_pack_data {\n> \tstruct object_array wants;\n> \tstruct oid_array haves;\n> \n> +\tstruct object_array shallows;\n> +\tstruct string_list deepen_not;\n> +\tint depth;\n> +\ttimestamp_t deepen_since;\n> +\tint deepen_rev_list;\n> +\tint deepen_relative;\n\nNice.\n\nComments describing deepen_Rev_list and deepen_relative would be nice.\n\nAre those boolean? Can they be unsigned:1 to make that\nself-explanatory?\n\n[...]\n> @@ -1080,12 +1101,14 @@ struct upload_pack_data {\n> \tunsigned done : 1;\n> };\n> \n> -#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }\n> +#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, OBJECT_ARRAY_INIT, STRING_LIST_INIT_DUP, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }\n\nLong line, \"make style\" should be able to fix it.\n\n[...]\n> \n> static void upload_pack_data_clear(struct upload_pack_data *data)\n> {\n> \tobject_array_clear(&data->wants);\n> \toid_array_clear(&data->haves);\n> +\tobject_array_clear(&data->shallows);\n> +\tstring_list_clear(&data->deepen_not, 0);\n\nThanks for remembering to clean up.\n\n[...]\n> @@ -1284,6 +1323,23 @@ static int process_haves_and_send_acks(struct upload_pack_data *data)\n> \treturn ret;\n> }\n> \n> +static void send_shallow_info(struct upload_pack_data *data)\n> +{\n> +\t/* No shallow info needs to be sent */\n> +\tif (!data->depth && !data->deepen_rev_list && !data->shallows.nr &&\n> +\t !is_repository_shallow())\n> +\t\treturn;\n> +\n> +\tpacket_write_fmt(1, \"shallow-info\\n\");\n> +\n> +\tif (!send_shallow_list(data->depth, data->deepen_rev_list,\n> +\t\t\t data->deepen_since, &data->deepen_not,\n> +\t\t\t &data->shallows) && is_repository_shallow())\n> +\t\tdeepen(INFINITE_DEPTH, data->deepen_relative, &data->shallows);\n> +\n> +\tpacket_delim(1);\n> +}\n\nNice.\n\n[...]\n> @@ -1346,3 +1404,11 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys,\n> \tupload_pack_data_clear(&data);\n> \treturn 0;\n> }\n> +\n> +int upload_pack_advertise(struct repository *r,\n> +\t\t\t struct strbuf *value)\n> +{\n> +\tif (value)\n> +\t\tstrbuf_addstr(value, \"shallow\");\n> +\treturn 1;\n> +}\n\nThis is about capabilities?\n\nMaybe a doc comment in the header file would help.\n\nThanks,\nJonathan\n"},{"id":"340421","messageId":"20180227183407.GD209668@google.com","threadId":"47524","inReplyTo":"20180227100555.91f2a5e196bff3968871a16e@google.com","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-27T18:34:07Z","receivedAt":"2018-02-27T18:34:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/27, Jonathan Tan wrote:\n> On Fri, 23 Feb 2018 13:33:15 -0800\n> Brandon Williams wrote:\n> \n> > On 02/21, Jonathan Tan wrote:\n> > > As someone who is implementing the server side of protocol V2 in JGit, I\n> > > now have a bit more insight into this :-)\n> > > \n> > > First of all, I used to not have a strong opinion on the existence of a\n> > > new endpoint, but now I think that it's better to *not* have git-serve.\n> > > As it is, as far as I can tell, upload-pack also needs to support (and\n> > > does support, as of the end of this patch set) protocol v2 anyway, so it\n> > > might be better to merely upgrade upload-pack.\n> > \n> > Having it allows for easier testing and the easy ability to make it a\n> > true endpoint when we want to. As of right now, git-serve isn't an\n> > endpoint as you can't issue requests there via http-backend or\n> > git-daemon.\n> \n> Is git-serve planned to be a new endpoint?\n> \n> If yes, I now don't think it's a good idea - it's an extra burden to\n> reimplementors without much benefit (to have a new endpoint that does\n> the same things as upload-pack).\n\nI'm still going to include it, with the potential for it to become an\nendpoint if we so choose (it isn't now), because when we start to\nintroduce more things to v2 (push or other commands we haven't dreamed\nup yet) it just makes more sense to contact an endpoint that doesn't\nexplicitly say what it does.\n\n> \n> If not, I don't think that easier testing makes it worth having an extra\n> binary. Couldn't the same tests be done by running upload-pack directly?\n\nits builtin and not a new binary, and yes it makes testing much easier\nbecause its assumes v2 from the start instead of v0.\n\n-- \nBrandon Williams\n"},{"id":"340422","messageId":"20180227184110.GE209668@google.com","threadId":"47524","inReplyTo":"20180227061813.GF65699@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 10/35] protocol: introduce enum protocol_version value protocol_v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-27T18:41:10Z","receivedAt":"2018-02-27T18:41:19Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/26, Jonathan Nieder wrote:\n> Hi,\n> \n> Brandon Williams wrote:\n> \n> > Introduce protocol_v2, a new value for 'enum protocol_version'.\n> > Subsequent patches will fill in the implementation of protocol_v2.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> \n> Yay!\n> \n> [...]\n> > +++ b/builtin/fetch-pack.c\n> > @@ -201,6 +201,9 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> > \t\t\t PACKET_READ_GENTLE_ON_EOF);\n> > \n> > \tswitch (discover_version(&reader)) {\n> > +\tcase protocol_v2:\n> > +\t\tdie(\"support for protocol v2 not implemented yet\");\n> > +\t\tbreak;\n> \n> This code goes away in a later patch, so no need to do anything about\n> this, but the 'break' is redundant after the 'die'.\n\nI'll fix that.\n\n> \n> [...]\n> > --- a/builtin/receive-pack.c\n> > +++ b/builtin/receive-pack.c\n> > @@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> > \t\tunpack_limit = receive_unpack_limit;\n> > \n> > \tswitch (determine_protocol_version_server()) {\n> > +\tcase protocol_v2:\n> > +\t\t/*\n> > +\t\t * push support for protocol v2 has not been implemented yet,\n> > +\t\t * so ignore the request to use v2 and fallback to using v0.\n> > +\t\t */\n> > +\t\tbreak;\n> \n> As you mentioned in the cover letter, it's probably worth doing the\n> same fallback on the client side (send-pack), too.\n> \n> Otherwise when this client talks to a new-enough server, it would\n> request protocol v2 and then get confused when the server responds\n> with the protocol v2 it requested.\n\nSome patches later on ensure this.\n\n> \n> Thanks,\n> Jonathan\n\n-- \nBrandon Williams\n"},{"id":"340425","messageId":"20180227185733.GF209668@google.com","threadId":"47524","inReplyTo":"20180227182854.GA174036@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 22/35] upload-pack: support shallow requests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-27T18:57:33Z","receivedAt":"2018-02-27T18:57:42Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/27, Jonathan Nieder wrote:\n\nI'll make the documentation changes you suggested.\n\n> > + deepen \n> > +\tRequest that the fetch/clone should be shallow having a commit depth of\n> \n> nit: s/Request/Requests/, for consistency with the others?\n> \n> > +\t relative to the remote side.\n> \n> What does the value of mean? E.g. does a depth of 1 mean to\n> fetch only the commits named in \"have\", 2 to fetch those commits plus\n> their parents, etc, or am I off by one?\n\nHonestly I have no clue, what does the current protocol do? There isn't\nany documentation about it and this just reuses the logic from that.\n\n> \n> Is always a positive number?\n> \n> What happens if starts with a 0? Is that a client error?\n> \n\n> > output = *section\n> > - section = (acknowledgments | packfile)\n> > + section = (acknowledgments | shallow-info | packfile)\n> > \t (flush-pkt | delim-pkt)\n> \n> It looks like sections can go in an arbitrary order. Are there\n> tests to make sure the server can cope with reordering? (I ask\n> not because I mistrust the server but because I have some vague\n> hope that other server implementations might be inspired by our\n> tests.)\n\nI'll fix this so that they don't come in arbitrary order\n\n> \n> [...]\n> > @@ -215,6 +245,11 @@ header.\n> > nak = PKT-LINE(\"NAK\" LF)\n> > ack = PKT-LINE(\"ACK\" SP obj-id LF)\n> > \n> > + shallow-info = PKT-LINE(\"shallow-info\" LF)\n> > +\t\t *PKT-LINE((shallow | unshallow) LF)\n> > + shallow = \"shallow\" SP obj-id\n> > + unshallow = \"unshallow\" SP obj-id\n> \n> Likewise: it looks like shallows and unshallows can be intermixed; can\n> this be either (a) tightened or (b) covered by tests to make sure a\n> later refactoring doesn't accidentally tighten it?\n\nThis reuses the existing logic from v0 so its due to that spec.\n\n> > --- a/upload-pack.c\n> > +++ b/upload-pack.c\n> > @@ -710,7 +710,6 @@ static void deepen(int depth, int deepen_relative,\n> > \t}\n> > \n> > \tsend_unshallow(shallows);\n> > -\tpacket_flush(1);\n> \n> What does this part do?\n> > }\n> > \n> > static void deepen_by_rev_list(int ac, const char **av,\n> > @@ -722,7 +721,52 @@ static void deepen_by_rev_list(int ac, const char **av,\n> > \tsend_shallow(result);\n> > \tfree_commit_list(result);\n> > \tsend_unshallow(shallows);\n> > -\tpacket_flush(1);\n> \n> Same question.\n\nPulling out the flush packet so that the logic can be reused for v2, the\nflush is added back in for the v0 case but not for the v2 case.\n\n> \n> > +}\n> > +\n> > +static int send_shallow_list(int depth, int deepen_rev_list,\n> > +\t\t\t timestamp_t deepen_since,\n> > +\t\t\t struct string_list *deepen_not,\n> > +\t\t\t struct object_array *shallows)\n> \n> What does the return value from this function represent? It doesn't\n> appear to be the usual \"0 means success, -1 means failure\" so a\n> comment would help.\n\nI'll add a comment.\n\n> \n> > +{\n> > +\tint ret = 0;\n> > +\n> > +\tif (depth > 0 && deepen_rev_list)\n> > +\t\tdie(\"git upload-pack: deepen and deepen-since (or deepen-not) cannot be used together\");\n> \n> nit: long line (can/should \"make style\" find these?)\n> \n> The error message is pretty long, longer than a typical 80-column\n> terminal, so probably best to find a way to make the message shorter.\n> E.g.\n> \n> \t\tdie(\"upload-pack: deepen cannot be combined with other deepen-* options\");\n> \n> That still would be >80 columns with the indent, so the usual style\n> would be to break it into multiple strings and use C preprocessor\n> concatenation (yuck):\n> \n> \t\tdie(\"upload-pack: \"\n> \t\t \"deepen cannot be combined with other deepen-* options\");\n> \n\n> [...]\n> > +\tif (depth > 0) {\n> > +\t\tdeepen(depth, deepen_relative, shallows);\n> > +\t\tret = 1;\n> > +\t} else if (deepen_rev_list) {\n> > +\t\tstruct argv_array av = ARGV_ARRAY_INIT;\n> > +\t\tint i;\n> > +\n> > +\t\targv_array_push(&av, \"rev-list\");\n> > +\t\tif (deepen_since)\n> > +\t\t\targv_array_pushf(&av, \"--max-age=%\"PRItime, deepen_since);\n> > +\t\tif (deepen_not->nr) {\n> > +\t\t\targv_array_push(&av, \"--not\");\n> > +\t\t\tfor (i = 0; i < deepen_not->nr; i++) {\n> > +\t\t\t\tstruct string_list_item *s = deepen_not->items + i;\n> > +\t\t\t\targv_array_push(&av, s->string);\n> \n> This accepts arbitrary rev-list arguments, which feels dangerous\n> (could end up doing an expensive operation or reading arbitrary files\n> or finding a way to execute arbitrary code).\n> \n> [...]\n> > -\t\tif (deepen_not.nr) {\n> > -\t\t\targv_array_push(&av, \"--not\");\n> > -\t\t\tfor (i = 0; i < deepen_not.nr; i++) {\n> > -\t\t\t\tstruct string_list_item *s = deepen_not.items + i;\n> > -\t\t\t\targv_array_push(&av, s->string);\n> \n> Huh. Looks like some of the above comments are better addressed to an\n> earlier patch.\n\nIf someone wants to fix this after the fact they can, I just moved this\nlogic, I didn't add it.\n\n> \n> [...]\n> > @@ -1071,6 +1085,13 @@ struct upload_pack_data {\n> > \tstruct object_array wants;\n> > \tstruct oid_array haves;\n> > \n> > +\tstruct object_array shallows;\n> > +\tstruct string_list deepen_not;\n> > +\tint depth;\n> > +\ttimestamp_t deepen_since;\n> > +\tint deepen_rev_list;\n> > +\tint deepen_relative;\n> \n> Nice.\n> \n> Comments describing deepen_Rev_list and deepen_relative would be nice.\n> \n> Are those boolean? Can they be unsigned:1 to make that\n> self-explanatory?\n\nThey are boolean but are passed via reference at some points so they\ncan't be bit flags.\n\n> \n> [...]\n> > @@ -1080,12 +1101,14 @@ struct upload_pack_data {\n> > \tunsigned done : 1;\n> > };\n> > \n> > -#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, 0, 0, 0, 0, 0, 0 }\n> > +#define UPLOAD_PACK_DATA_INIT { OBJECT_ARRAY_INIT, OID_ARRAY_INIT, OBJECT_ARRAY_INIT, STRING_LIST_INIT_DUP, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }\n> \n> Long line, \"make style\" should be able to fix it.\n> \n\nI'll fix this.\n\n-- \nBrandon Williams\n"},{"id":"340427","messageId":"20180227192024.GB174036@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180227181428.GB209668@google.com","subject":"Re: [PATCH v3 02/35] pkt-line: introduce struct packet_reader","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T19:20:24Z","receivedAt":"2018-02-27T19:20:32Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n> On 02/12, Jonathan Nieder wrote:\n\n>>> --- a/pkt-line.h\n>>> +++ b/pkt-line.h\n>>> @@ -111,6 +111,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n>>> */\n>>> ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);\n>>> \n>>> +struct packet_reader {\n>>> +\t/* source file descriptor */\n>>> +\tint fd;\n>>> +\n>>> +\t/* source buffer and its size */\n>>> +\tchar *src_buffer;\n>>> +\tsize_t src_len;\n>>\n>> Can or should this be a strbuf?\n>>\n>>> +\n>>> +\t/* buffer that pkt-lines are read into and its size */\n>>> +\tchar *buffer;\n>>> +\tunsigned buffer_size;\n>>\n>> Likewise.\n>\n> This struct is setup to be a drop in replacement for the existing\n> read_packet() family of functions. Because of this I tried to make the\n> interface as similar as possible to make it easy to convert to using it\n> as well as having no need to clean anything up (because the struct is\n> really just a wrapper and doesn't own anything).\n\nSorry, I don't completely follow. Are you saying some callers play\nwith the buffer, or are you saying you haven't checked? (If the\nlatter, that's perfectly fine; I'm just trying to understand the API.)\n\nEither way, can you add some comments about ownership / who is allowed\nto write to it / etc to make it easier to clean up later?\n\nThanks,\nJonathan\n"},{"id":"340428","messageId":"20180227192514.GC174036@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180227181715.GC209668@google.com","subject":"Re: [PATCH v3 06/35] transport: use get_refs_via_connect to get refs","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T19:25:14Z","receivedAt":"2018-02-27T19:25:23Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n> On 02/26, Jonathan Nieder wrote:\n>> Brandon Williams wrote:\n\n>>> +++ b/transport.c\n>>> @@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n>>> \targs.cloning = transport->cloning;\n>>> \targs.update_shallow = data->options.update_shallow;\n>>> \n>>> -\tif (!data->got_remote_heads) {\n>>> -\t\tconnect_setup(transport, 0);\n>>> -\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n>>> -\t\t\t\t NULL, &data->shallow);\n>>> -\t\tdata->got_remote_heads = 1;\n>>> -\t}\n>>> +\tif (!data->got_remote_heads)\n>>> +\t\trefs_tmp = get_refs_via_connect(transport, 0);\n>>\n>> The only difference between the old and new code is that the old code\n>> passes NULL as 'extra_have' and the new code passes &data->extra_have.\n>>\n>> That means this populates the data->extra_have oid_array. Does it\n>> matter?\n[...]\n> I don't think its a problem to have extra_have populated, least I\n> haven't seen anything to lead me to believe it would be a problem.\n\nAssuming it gets properly freed later, the only effect I can imagine\nis some increased memory usage.\n\nI'm inclined to agree with you that the simplicity is worth it. It\nseems worth mentioning in the commit message, though.\n\n[...]\n>>> @@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n>>> \tstruct send_pack_args args;\n>>> \tint ret;\n>>> \n>>> -\tif (!data->got_remote_heads) {\n>>> -\t\tstruct ref *tmp_refs;\n>>> -\t\tconnect_setup(transport, 1);\n>>> -\n>>> -\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n>>> -\t\t\t\t NULL, &data->shallow);\n>>> -\t\tdata->got_remote_heads = 1;\n>>> -\t}\n>>> +\tif (!data->got_remote_heads)\n>>> +\t\tget_refs_via_connect(transport, 1);\n>>\n>> not a new problem, just curious: Does this leak tmp_refs?\n>\n> Maybe, though its removed by this patch.\n\nSorry for the lack of clarity. If it was leaked before, then it is\nstill leaked now, via the discarded return value from\nget_refs_via_connect.\n\nAny idea how we can track that down? E.g. are there ways to tell leak\ncheckers \"just tell me about this particular allocation\"?\n\nThanks,\nJonathan\n"},{"id":"340429","messageId":"CAGZ79kZLkHwfiqSf8htKPRKyVBCvurQwTAwW4n87ESbBfaL3sA@mail.gmail.com","threadId":"47524","inReplyTo":"20180207011312.189834-22-bmwill@google.com","subject":"Re: [PATCH v3 21/35] fetch-pack: perform a fetch using v2","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-27T19:27:25Z","receivedAt":"2018-02-27T19:27:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> When communicating with a v2 server, perform a fetch by requesting the\n> 'fetch' command.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> builtin/fetch-pack.c | 2 +-\n> fetch-pack.c | 252 ++++++++++++++++++++++++++++++++++++++++++++++++-\n> fetch-pack.h | 4 +-\n> t/t5702-protocol-v2.sh | 84 +++++++++++++++++\n> transport.c | 7 +-\n> 5 files changed, 342 insertions(+), 7 deletions(-)\n>\n> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n> index f492e8abd..867dd3cc7 100644\n> --- a/builtin/fetch-pack.c\n> +++ b/builtin/fetch-pack.c\n> @@ -213,7 +213,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> }\n>\n> ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n> - &shallow, pack_lockfile_ptr);\n> + &shallow, pack_lockfile_ptr, protocol_v0);\n> if (pack_lockfile) {\n> printf(\"lock %s\\n\", pack_lockfile);\n> fflush(stdout);\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 9f6b07ad9..4fb5805dd 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -1008,6 +1008,247 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n> return ref;\n> }\n>\n> +static void add_wants(const struct ref *wants, struct strbuf *req_buf)\n> +{\n> + for ( ; wants ; wants = wants->next) {\n> + const struct object_id *remote = &wants->old_oid;\n> + const char *remote_hex;\n> + struct object *o;\n> +\n> + /*\n> + * If that object is complete (i.e. it is an ancestor of a\n> + * local ref), we tell them we have it but do not have to\n> + * tell them about its ancestors, which they already know\n> + * about.\n> + *\n> + * We use lookup_object here because we are only\n> + * interested in the case we *know* the object is\n> + * reachable and we have already scanned it.\n> + */\n> + if (((o = lookup_object(remote->hash)) != NULL) &&\n> + (o->flags & COMPLETE)) {\n> + continue;\n> + }\n> +\n> + remote_hex = oid_to_hex(remote);\n> + packet_buf_write(req_buf, \"want %s\\n\", remote_hex);\n> + }\n> +}\n> +\n> +static void add_common(struct strbuf *req_buf, struct oidset *common)\n> +{\n> + struct oidset_iter iter;\n> + const struct object_id *oid;\n> + oidset_iter_init(common, &iter);\n> +\n> + while ((oid = oidset_iter_next(&iter))) {\n> + packet_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n> + }\n> +}\n> +\n> +static int add_haves(struct strbuf *req_buf, int *in_vain)\n> +{\n> + int ret = 0;\n> + int haves_added = 0;\n> + const struct object_id *oid;\n> +\n> + while ((oid = get_rev())) {\n> + packet_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n> + if (++haves_added >= INITIAL_FLUSH)\n> + break;\n> + };\n> +\n> + *in_vain += haves_added;\n> + if (!haves_added || *in_vain >= MAX_IN_VAIN) {\n> + /* Send Done */\n> + packet_buf_write(req_buf, \"done\\n\");\n> + ret = 1;\n> + }\n> +\n> + return ret;\n> +}\n> +\n> +static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n> + const struct ref *wants, struct oidset *common,\n> + int *in_vain)\n> +{\n> + int ret = 0;\n> + struct strbuf req_buf = STRBUF_INIT;\n> +\n> + if (server_supports_v2(\"fetch\", 1))\n> + packet_buf_write(&req_buf, \"command=fetch\");\n> + if (server_supports_v2(\"agent\", 0))\n> + packet_buf_write(&req_buf, \"agent=%s\", git_user_agent_sanitized());\n> +\n> + packet_buf_delim(&req_buf);\n> + if (args->use_thin_pack)\n> + packet_buf_write(&req_buf, \"thin-pack\");\n> + if (args->no_progress)\n> + packet_buf_write(&req_buf, \"no-progress\");\n> + if (args->include_tag)\n> + packet_buf_write(&req_buf, \"include-tag\");\n> + if (prefer_ofs_delta)\n> + packet_buf_write(&req_buf, \"ofs-delta\");\n> +\n> + /* add wants */\n> + add_wants(wants, &req_buf);\n\nThe comment might convey too much redundant information\ninstead of helping the reader gain more understanding. ;)\n\n> +\n> + /* Add all of the common commits we've found in previous rounds */\n> + add_common(&req_buf, common);\n\nnit:\nMaybe s/add_common/add_common_haves/ or\nadd_previous_haves ?\n\n> +\n> + /* Add initial haves */\n> + ret = add_haves(&req_buf, in_vain);\n\nI like the shortness and conciseness of this send_fetch_request\nfunction as it makes clear what is happening over the wire, however\nI wonder if we can improve on that, still.\n\nThe functions 'add_common' and 'add_haves' seem like they do the\nsame (sending haves), except for different sets of oids.\n\nSo I would imagine that a structure like\n\n {\n struct set haves = compute_haves_from(in_vain, common, ...);\n struct set wants = compute_wants&wants);\n\n request_capabilities(args)\n send_haves(&haves);\n send_wants(&wants);\n flush();\n }\n\nThat way we would have an even more concise way of writing\none request, and factoring out the business logic. (Coming up\nwith the \"right\" haves is a heuristic that we plan on changing in\nthe future, so we'd want to have that encapsulated into one function\nthat computes all the haves?\n\n> +\n> +/*\n> + * Processes a section header in a server's response and checks if it matches\n> + * `section`. If the value of `peek` is 1, the header line will be peeked (and\n> + * not consumed); if 0, the line will be consumed and the function will die if\n> + * the section header doesn't match what was expected.\n> + */\n> +static int process_section_header(struct packet_reader *reader,\n> + const char *section, int peek)\n> +{\n> + int ret;\n> +\n> + if (packet_reader_peek(reader) != PACKET_READ_NORMAL)\n> + die(\"error reading packet\");\n> +\n> + ret = !strcmp(reader->line, section);\n> +\n> + if (!peek) {\n> + if (!ret)\n> + die(\"expected '%s', received '%s'\",\n> + section, reader->line);\n> + packet_reader_read(reader);\n> + }\n> +\n> + return ret;\n> +}\n> +\n> +static int process_acks(struct packet_reader *reader, struct oidset *common)\n> +{\n> + /* received */\n> + int received_ready = 0;\n> + int received_ack = 0;\n> +\n> + process_section_header(reader, \"acknowledgments\", 0);\n> + while (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n> + const char *arg;\n> +\n> + if (!strcmp(reader->line, \"NAK\"))\n> + continue;\n> +\n> + if (skip_prefix(reader->line, \"ACK \", &arg)) {\n> + struct object_id oid;\n> + if (!get_oid_hex(arg, &oid)) {\n> + struct commit *commit;\n> + oidset_insert(common, &oid);\n> + commit = lookup_commit(&oid);\n> + mark_common(commit, 0, 1);\n> + }\n> + continue;\n> + }\n> +\n> + if (!strcmp(reader->line, \"ready\")) {\n> + clear_prio_queue(&rev_list);\n> + received_ready = 1;\n> + continue;\n> + }\n> +\n> + die(_(\"git fetch-pack: expected ACK/NAK, got '%s'\"), reader->line);\n\nThis is slightly misleading, it could also expect \"ready\" ?\n\n\n> + }\n> +\n> + if (reader->status != PACKET_READ_FLUSH &&\n> + reader->status != PACKET_READ_DELIM)\n> + die(\"Error during processing acks: %d\", reader->status);\n\nWhy is this not translated unlike the one 5 lines prior to this?\nDo we expect these conditions to come up due to different\nroot causes?\n\n> +static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> + int fd[2],\n> + const struct ref *orig_ref,\n> + struct ref **sought, int nr_sought,\n> + char **pack_lockfile)\n> +{\n> + struct ref *ref = copy_ref_list(orig_ref);\n> + enum fetch_state state = FETCH_CHECK_LOCAL;\n> + struct oidset common = OIDSET_INIT;\n> + struct packet_reader reader;\n> + int in_vain = 0;\n> + packet_reader_init(&reader, fd[0], NULL, 0,\n> + PACKET_READ_CHOMP_NEWLINE);\n> +\n> + while (state != FETCH_DONE) {\n> + switch (state) {\n> + case FETCH_CHECK_LOCAL:\n> + sort_ref_list(&ref, ref_compare_name);\n> + QSORT(sought, nr_sought, cmp_ref_by_name);\n> +\n> + /* v2 supports these by default */\n\nIs there a doc that says what is all on by default?\n\nThanks,\nStefan\n"},{"id":"340430","messageId":"20180227193023.GG209668@google.com","threadId":"47524","inReplyTo":"20180227065158.GH65699@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-27T19:30:23Z","receivedAt":"2018-02-27T19:30:31Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/26, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> > static char *server_capabilities;\n> > +static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n> \n> Can a quick doc comment describe these and how they relate?\n> \n> Is only one of them set, based on which protocol version is in use?\n> Should server_capabilities be renamed to server_capabilities_v1?\n\nyes that's correct. I can rename it.\n\n> > +{\n> > +\tint ret = 1;\n> > +\tint i = 0;\n> > +\tstruct object_id old_oid;\n> > +\tstruct ref *ref;\n> > +\tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n> > +\n> > +\tif (string_list_split(&line_sections, line, ' ', -1) < 2) {\n> \n> Can there be a comment describing the expected format?\n\nYep I'll write a comment up.\n\n> > +\n> > +\toidcpy(&ref->old_oid, &old_oid);\n> > +\t**list = ref;\n> > +\t*list = &ref->next;\n> > +\n> > +\tfor (; i < line_sections.nr; i++) {\n> > +\t\tconst char *arg = line_sections.items[i].string;\n> > +\t\tif (skip_prefix(arg, \"symref-target:\", &arg))\n> > +\t\t\tref->symref = xstrdup(arg);\n> \n> Using space-delimited fields in a single pkt-line means that\n> - values cannot contains a space\n> - total length is limited by the size of a pkt-line\n> \n> Given the context, I think that's fine. More generally it is tempting\n> to use a pkt-line per field to avoid the trouble v1 had with\n> capability lists crammed into a pkt-line, but I see why you used a\n> pkt-line per ref to avoid having to have sections-within-a-section.\n> \n> My only potential worry is the length part: do we have an explicit\n> limit on the length of a ref name? git-check-ref-format(1) doesn't\n> mention one. A 32k ref name would be a bit ridiculous, though.\n\nYeah I think we're fine for now, mostly because we're out of luck with\nthe current protocol as it is.\n\n> \n> > +\n> > +\t\tif (skip_prefix(arg, \"peeled:\", &arg)) {\n> > +\t\t\tstruct object_id peeled_oid;\n> > +\t\t\tchar *peeled_name;\n> > +\t\t\tstruct ref *peeled;\n> > +\t\t\tif (get_oid_hex(arg, &peeled_oid)) {\n> > +\t\t\t\tret = 0;\n> > +\t\t\t\tgoto out;\n> > +\t\t\t}\n> \n> Can this also check that there's no trailing garbage after the oid?\n\nYeah I do that.\n\n> \n> > +\n> > +\tpacket_delim(fd_out);\n> > +\t/* When pushing we don't want to request the peeled tags */\n> \n> Can you say more about this? In theory it would be nice to have the\n> peeled tags since they name commits whose history can be excluded from\n> the pack.\n\nI don't believe peeled refs are sent now in v0 for push.\n\n> \n> > +\tif (!for_push)\n> > +\t\tpacket_write_fmt(fd_out, \"peel\\n\");\n> > +\tpacket_write_fmt(fd_out, \"symrefs\\n\");\n> \n> Are symrefs useful during push?\n\nThey may be at a later point in time when you want to update a symref :)\n\n> \n> > +\tfor (i = 0; ref_patterns && i < ref_patterns->argc; i++) {\n> > +\t\tpacket_write_fmt(fd_out, \"ref-pattern %s\\n\",\n> > +\t\t\t\t ref_patterns->argv[i]);\n> > +\t}\n> \n> The exciting part.\n> \n> Why do these pkts end with \\n? I would have expected the pkt-line\n> framing to work to delimit them.\n\nAll pkts end with \\n, that's just hows its been since v0. Though the\nserver isn't supposed to complain if they don't contain newlines.\n\n> \n> > +\tpacket_flush(fd_out);\n> > +\n> > +\t/* Process response from server */\n> > +\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n> > +\t\tif (!process_ref_v2(reader->line, &list))\n> > +\t\t\tdie(\"invalid ls-refs response: %s\", reader->line);\n> > +\t}\n> > +\n> > +\tif (reader->status != PACKET_READ_FLUSH)\n> > +\t\tdie(\"protocol error\");\n> \n> Can this protocol error give more detail? When diagnosing an error in\n> servers, proxies, or lower-level networking issues, informative protocol\n> errors can be very helpful (similar to syntax errors from a compiler).\n\nI'll update the error msg.\n\n> [...]\n> > --- a/remote.h\n> > +++ b/remote.h\n> > @@ -151,10 +151,14 @@ void free_refs(struct ref *ref);\n> > \n> > struct oid_array;\n> > struct packet_reader;\n> > +struct argv_array;\n> > extern struct ref **get_remote_heads(struct packet_reader *reader,\n> > \t\t\t\t struct ref **list, unsigned int flags,\n> > \t\t\t\t struct oid_array *extra_have,\n> > \t\t\t\t struct oid_array *shallow_points);\n> > +extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n> > +\t\t\t\t struct ref **list, int for_push,\n> > +\t\t\t\t const struct argv_array *ref_patterns);\n> \n> What is the difference between get_remote_heads and get_remote_refs?\n> A comment might help. (BTW, thanks for making the new saner name to\n> replace get_remote_heads!)\n\nI'll add a comment saying its used in v2 to retrieve a list of refs from\nthe remote.\n\n-- \nBrandon Williams\n"},{"id":"340431","messageId":"20180227194047.GH209668@google.com","threadId":"47524","inReplyTo":"CAGZ79kZLkHwfiqSf8htKPRKyVBCvurQwTAwW4n87ESbBfaL3sA@mail.gmail.com","subject":"Re: [PATCH v3 21/35] fetch-pack: perform a fetch using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-27T19:40:47Z","receivedAt":"2018-02-27T19:40:55Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/27, Stefan Beller wrote:\n> > +\n> > + /* Add initial haves */\n> > + ret = add_haves(&req_buf, in_vain);\n> \n> I like the shortness and conciseness of this send_fetch_request\n> function as it makes clear what is happening over the wire, however\n> I wonder if we can improve on that, still.\n\nI'm sure there is more that can be micro optimized but I don't really\nwant to get distracted by redesigning this logic another time right now.\n\n> \n> The functions 'add_common' and 'add_haves' seem like they do the\n> same (sending haves), except for different sets of oids.\n> \n> So I would imagine that a structure like\n> \n> {\n> struct set haves = compute_haves_from(in_vain, common, ...);\n> struct set wants = compute_wants&wants);\n> \n> request_capabilities(args)\n> send_haves(&haves);\n> send_wants(&wants);\n> flush();\n> }\n> \n> That way we would have an even more concise way of writing\n> one request, and factoring out the business logic. (Coming up\n> with the \"right\" haves is a heuristic that we plan on changing in\n> the future, so we'd want to have that encapsulated into one function\n> that computes all the haves?\n> \n> > +\n> > +/*\n> > + * Processes a section header in a server's response and checks if it matches\n> > + * `section`. If the value of `peek` is 1, the header line will be peeked (and\n> > + * not consumed); if 0, the line will be consumed and the function will die if\n> > + * the section header doesn't match what was expected.\n> > + */\n> > +static int process_section_header(struct packet_reader *reader,\n> > + const char *section, int peek)\n> > +{\n> > + int ret;\n> > +\n> > + if (packet_reader_peek(reader) != PACKET_READ_NORMAL)\n> > + die(\"error reading packet\");\n> > +\n> > + ret = !strcmp(reader->line, section);\n> > +\n> > + if (!peek) {\n> > + if (!ret)\n> > + die(\"expected '%s', received '%s'\",\n> > + section, reader->line);\n> > + packet_reader_read(reader);\n> > + }\n> > +\n> > + return ret;\n> > +}\n> > +\n> > +static int process_acks(struct packet_reader *reader, struct oidset *common)\n> > +{\n> > + /* received */\n> > + int received_ready = 0;\n> > + int received_ack = 0;\n> > +\n> > + process_section_header(reader, \"acknowledgments\", 0);\n> > + while (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n> > + const char *arg;\n> > +\n> > + if (!strcmp(reader->line, \"NAK\"))\n> > + continue;\n> > +\n> > + if (skip_prefix(reader->line, \"ACK \", &arg)) {\n> > + struct object_id oid;\n> > + if (!get_oid_hex(arg, &oid)) {\n> > + struct commit *commit;\n> > + oidset_insert(common, &oid);\n> > + commit = lookup_commit(&oid);\n> > + mark_common(commit, 0, 1);\n> > + }\n> > + continue;\n> > + }\n> > +\n> > + if (!strcmp(reader->line, \"ready\")) {\n> > + clear_prio_queue(&rev_list);\n> > + received_ready = 1;\n> > + continue;\n> > + }\n> > +\n> > + die(_(\"git fetch-pack: expected ACK/NAK, got '%s'\"), reader->line);\n> \n> This is slightly misleading, it could also expect \"ready\" ?\n\nI'll update this.\n\n> \n> \n> > + }\n> > +\n> > + if (reader->status != PACKET_READ_FLUSH &&\n> > + reader->status != PACKET_READ_DELIM)\n> > + die(\"Error during processing acks: %d\", reader->status);\n> \n> Why is this not translated unlike the one 5 lines prior to this?\n> Do we expect these conditions to come up due to different\n> root causes?\n> \n> > +static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> > + int fd[2],\n> > + const struct ref *orig_ref,\n> > + struct ref **sought, int nr_sought,\n> > + char **pack_lockfile)\n> > +{\n> > + struct ref *ref = copy_ref_list(orig_ref);\n> > + enum fetch_state state = FETCH_CHECK_LOCAL;\n> > + struct oidset common = OIDSET_INIT;\n> > + struct packet_reader reader;\n> > + int in_vain = 0;\n> > + packet_reader_init(&reader, fd[0], NULL, 0,\n> > + PACKET_READ_CHOMP_NEWLINE);\n> > +\n> > + while (state != FETCH_DONE) {\n> > + switch (state) {\n> > + case FETCH_CHECK_LOCAL:\n> > + sort_ref_list(&ref, ref_compare_name);\n> > + QSORT(sought, nr_sought, cmp_ref_by_name);\n> > +\n> > + /* v2 supports these by default */\n> \n> Is there a doc that says what is all on by default?\n\nYeah protocol-v2.txt should say all of that.\n\n> \n> Thanks,\n> Stefan\n\n-- \nBrandon Williams\n"},{"id":"340432","messageId":"20180227194604.GI209668@google.com","threadId":"47524","inReplyTo":"20180227192514.GC174036@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 06/35] transport: use get_refs_via_connect to get refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-27T19:46:04Z","receivedAt":"2018-02-27T19:46:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/27, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> > On 02/26, Jonathan Nieder wrote:\n> >> Brandon Williams wrote:\n> \n> >>> +++ b/transport.c\n> >>> @@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n> >>> \targs.cloning = transport->cloning;\n> >>> \targs.update_shallow = data->options.update_shallow;\n> >>> \n> >>> -\tif (!data->got_remote_heads) {\n> >>> -\t\tconnect_setup(transport, 0);\n> >>> -\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n> >>> -\t\t\t\t NULL, &data->shallow);\n> >>> -\t\tdata->got_remote_heads = 1;\n> >>> -\t}\n> >>> +\tif (!data->got_remote_heads)\n> >>> +\t\trefs_tmp = get_refs_via_connect(transport, 0);\n> >>\n> >> The only difference between the old and new code is that the old code\n> >> passes NULL as 'extra_have' and the new code passes &data->extra_have.\n> >>\n> >> That means this populates the data->extra_have oid_array. Does it\n> >> matter?\n> [...]\n> > I don't think its a problem to have extra_have populated, least I\n> > haven't seen anything to lead me to believe it would be a problem.\n> \n> Assuming it gets properly freed later, the only effect I can imagine\n> is some increased memory usage.\n> \n> I'm inclined to agree with you that the simplicity is worth it. It\n> seems worth mentioning in the commit message, though.\n> \n> [...]\n> >>> @@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n> >>> \tstruct send_pack_args args;\n> >>> \tint ret;\n> >>> \n> >>> -\tif (!data->got_remote_heads) {\n> >>> -\t\tstruct ref *tmp_refs;\n> >>> -\t\tconnect_setup(transport, 1);\n> >>> -\n> >>> -\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n> >>> -\t\t\t\t NULL, &data->shallow);\n> >>> -\t\tdata->got_remote_heads = 1;\n> >>> -\t}\n> >>> +\tif (!data->got_remote_heads)\n> >>> +\t\tget_refs_via_connect(transport, 1);\n> >>\n> >> not a new problem, just curious: Does this leak tmp_refs?\n> >\n> > Maybe, though its removed by this patch.\n> \n> Sorry for the lack of clarity. If it was leaked before, then it is\n> still leaked now, via the discarded return value from\n> get_refs_via_connect.\n> \n> Any idea how we can track that down? E.g. are there ways to tell leak\n> checkers \"just tell me about this particular allocation\"?\n\nHmm I wonder if that code path is even used, because it just throws away\nthe result.\n\n-- \nBrandon Williams\n"},{"id":"340464","messageId":"xmqq8tbep02f.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180227062128.GG65699@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-27T21:58:00Z","receivedAt":"2018-02-27T21:58:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> Jonathan Tan wrote:\n>> On Thu, 22 Feb 2018 13:26:58 -0500\n>> Jeff King wrote:\n>\n>>> I agree that it shouldn't matter much here. But if the name argv_array\n>>> is standing in the way of using it, I think we should consider giving it\n>>> a more general name. I picked that not to evoke \"this must be arguments\"\n>>> but \"this is terminated by a single NULL\".\n> [...]\n>> This sounds reasonable - I withdraw my comment about using struct\n>> string_list.\n>\n> Marking with #leftoverbits as a reminder to think about what such a\n> more general name would be (or what kind of docs to put in\n> argv-array.h) and make it so the next time I do a search for that\n> keyword.\n\nSo are we looking for a natural name to call an array of trings? I\npersonally do not mind argv_array too much, but perhaps we can call\nit a string_array and then everybody will be happy?\n"},{"id":"340466","messageId":"20180227220443.GB11187@sigill.intra.peff.net","threadId":"47524","inReplyTo":"xmqq8tbep02f.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-27T22:04:43Z","receivedAt":"2018-02-27T22:04:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 27, 2018 at 01:58:00PM -0800, Junio C Hamano wrote:\n\n> Jonathan Nieder writes:\n> \n> > Jonathan Tan wrote:\n> >> On Thu, 22 Feb 2018 13:26:58 -0500\n> >> Jeff King wrote:\n> >\n> >>> I agree that it shouldn't matter much here. But if the name argv_array\n> >>> is standing in the way of using it, I think we should consider giving it\n> >>> a more general name. I picked that not to evoke \"this must be arguments\"\n> >>> but \"this is terminated by a single NULL\".\n> > [...]\n> >> This sounds reasonable - I withdraw my comment about using struct\n> >> string_list.\n> >\n> > Marking with #leftoverbits as a reminder to think about what such a\n> > more general name would be (or what kind of docs to put in\n> > argv-array.h) and make it so the next time I do a search for that\n> > keyword.\n> \n> So are we looking for a natural name to call an array of trings? I\n> personally do not mind argv_array too much, but perhaps we can call\n> it a string_array and then everybody will be happy?\n\nThat would be fine with me. Though I would love it if we could find a\nshorter name for the associated functions. For example,\nargv_array_pushf() can make lines quite long, and something like\nargv_pushf() is easier to read (in my opinion). And that might work\nbecause \"argv\" is pretty unique by itself, but \"string\" is not.\n\nSome one-word name like \"strarray\" might work, though I find that is not\nquite catchy. I guess \"strv\" is short if you assume that people know the\n\"v\" suffix means \"vector\".\n\nIt may not be worth worrying too much about, though. We already have\n24-character monstrosities like string_list_append_nodup(). ;)\n\n-Peff\n"},{"id":"340471","messageId":"CAPig+cS+G-xC51n-Ud0Wbmcc-zeHBM3-5WQQAFm9gwm9LNk3Gg@mail.gmail.com","threadId":"47524","inReplyTo":"20180227220443.GB11187@sigill.intra.peff.net","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-02-27T22:10:09Z","receivedAt":"2018-02-27T22:10:29Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Feb 27, 2018 at 5:04 PM, Jeff King wrote:\n> On Tue, Feb 27, 2018 at 01:58:00PM -0800, Junio C Hamano wrote:\n>> So are we looking for a natural name to call an array of trings? I\n>> personally do not mind argv_array too much, but perhaps we can call\n>> it a string_array and then everybody will be happy?\n>\n> That would be fine with me. Though I would love it if we could find a\n> shorter name for the associated functions. For example,\n> argv_array_pushf() can make lines quite long, and something like\n> argv_pushf() is easier to read (in my opinion). And that might work\n> because \"argv\" is pretty unique by itself, but \"string\" is not.\n>\n> Some one-word name like \"strarray\" might work, though I find that is not\n> quite catchy. I guess \"strv\" is short if you assume that people know the\n> \"v\" suffix means \"vector\".\n\nstruct strs {...};\n\nvoid strs_init(struct strs *);\nvoid strs_push(struct strs *, const char *);\nvoid strs_pushf(struct strs *, const char *fmt, ...);\nvoid strs_pushl(struct strs *, ...);\nvoid strs_pushv(struct strs *, const char **);\nvoid strs_pop(struct strs *);\nvoid strs_clear(struct strs *);\nconst char **strs_detach(struct strs *);\n\n...is short, feels pretty natural, and doesn't require understanding\n\"v\" for \"vector\".\n"},{"id":"340474","messageId":"20180227221808.GE11187@sigill.intra.peff.net","threadId":"47524","inReplyTo":"CAPig+cS+G-xC51n-Ud0Wbmcc-zeHBM3-5WQQAFm9gwm9LNk3Gg@mail.gmail.com","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-27T22:18:08Z","receivedAt":"2018-02-27T22:18:15Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Feb 27, 2018 at 05:10:09PM -0500, Eric Sunshine wrote:\n\n> > That would be fine with me. Though I would love it if we could find a\n> > shorter name for the associated functions. For example,\n> > argv_array_pushf() can make lines quite long, and something like\n> > argv_pushf() is easier to read (in my opinion). And that might work\n> > because \"argv\" is pretty unique by itself, but \"string\" is not.\n> >\n> > Some one-word name like \"strarray\" might work, though I find that is not\n> > quite catchy. I guess \"strv\" is short if you assume that people know the\n> > \"v\" suffix means \"vector\".\n> \n> struct strs {...};\n> \n> void strs_init(struct strs *);\n> void strs_push(struct strs *, const char *);\n> void strs_pushf(struct strs *, const char *fmt, ...);\n> void strs_pushl(struct strs *, ...);\n> void strs_pushv(struct strs *, const char **);\n> void strs_pop(struct strs *);\n> void strs_clear(struct strs *);\n> const char **strs_detach(struct strs *);\n> \n> ...is short, feels pretty natural, and doesn't require understanding\n> \"v\" for \"vector\".\n\nNot bad. The \"v\" carries the information that it _is_ a NULL-terminated\nvector and not some other list-like structure (and so is suitable for\nfeeding to execv, etc). But that may just be obvious from looking at its\nuses and documentation.\n\n-Peff\n"},{"id":"340481","messageId":"20180227230316.GE174036@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-27-bmwill@google.com","subject":"Re: [PATCH v3 26/35] transport-helper: remove name parameter","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T23:03:16Z","receivedAt":"2018-02-27T23:03:27Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Commit 266f1fdfa (transport-helper: be quiet on read errors from\n> helpers, 2013-06-21) removed a call to 'die()' which printed the name of\n> the remote helper passed in to the 'recvline_fh()' function using the\n> 'name' parameter. Once the call to 'die()' was removed the parameter\n> was no longer necessary but wasn't removed. Clean up 'recvline_fh()'\n> parameter list by removing the 'name' parameter.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> transport-helper.c | 6 +++---\n> 1 file changed, 3 insertions(+), 3 deletions(-)\n\nNice.\n\nReviewed-by: Jonathan Nieder \n"},{"id":"340482","messageId":"20180227231145.GF174036@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-30-bmwill@google.com","subject":"Re: [PATCH v3 29/35] pkt-line: add packet_buf_write_len function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T23:11:45Z","receivedAt":"2018-02-27T23:11:54Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Add the 'packet_buf_write_len()' function which allows for writing an\n> arbitrary length buffer into a 'struct strbuf' and formatting it in\n> packet-line format.\n\nMakes sense.\n\n[...]\n> --- a/pkt-line.h\n> +++ b/pkt-line.h\n> @@ -26,6 +26,7 @@ void packet_buf_flush(struct strbuf *buf);\n> void packet_buf_delim(struct strbuf *buf);\n> void packet_write(int fd_out, const char *buf, size_t size);\n> void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n> +void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len);\n\nI wonder if we should rename packet_buf_write to something like\npacket_buf_writef. Right now there's a kind of confusing collection of\nfunctions without much symmetry.\n\nAlternatively, the _buf_ ones could become strbuf_* functions:\n\n\tstrbuf_add_packet(&buf, data, len);\n\tstrbuf_addf_packet(&buf, fmt, ...);\n\nThat would make it clearer that these append to buf.\n\nI'm just thinking out loud. For this series, the API you have here\nlooks fine, even if it is a bit inconsistent. (In other words, even\nif you agree with me, this would probably be best addressed as a patch\non top.)\n\n[...]\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -215,6 +215,22 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n> \tva_end(args);\n> }\n> \n> +void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len)\n> +{\n> +\tsize_t orig_len, n;\n> +\n> +\torig_len = buf->len;\n> +\tstrbuf_addstr(buf, \"0000\");\n> +\tstrbuf_add(buf, data, len);\n> +\tn = buf->len - orig_len;\n> +\n> +\tif (n > LARGE_PACKET_MAX)\n> +\t\tdie(\"protocol error: impossibly long line\");\n\nCould the error message describe the long line (e.g.\n\n\t\t...impossibly long line %.*s...\", 256, data);\n\n)?\n\n> +\n> +\tset_packet_header(&buf->buf[orig_len], n);\n> +\tpacket_trace(buf->buf + orig_len + 4, n - 4, 1);\n\nCould do, more simply:\n\n\tpacket_trace(data, len, 1);\n\nThanks,\nJonathan\n"},{"id":"340484","messageId":"20180227231749.GG174036@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-32-bmwill@google.com","subject":"Re: [PATCH v3 31/35] remote-curl: store the protocol version the server responded with","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T23:17:49Z","receivedAt":"2018-02-27T23:17:58Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Store the protocol version the server responded with when performing\n> discovery. This will be used in a future patch to either change the\n> 'Git-Protocol' header sent in subsequent requests or to determine if a\n> client needs to fallback to using a different protocol version.\n\nnit: s/fallback/fall back/ (fallback is the noun/adjective, fall back\nthe verb)\n\n> Signed-off-by: Brandon Williams \n\nWith or without that tweak,\nReviewed-by: Jonathan Nieder \n\nThanks.\n"},{"id":"340487","messageId":"20180227233025.GH174036@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-29-bmwill@google.com","subject":"Re: [PATCH v3 28/35] transport-helper: introduce stateless-connect","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-27T23:30:25Z","receivedAt":"2018-02-27T23:30:33Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n\n> Introduce the transport-helper capability 'stateless-connect'. This\n> capability indicates that the transport-helper can be requested to run\n> the 'stateless-connect' command which should attempt to make a\n> stateless connection with a remote end. Once established, the\n> connection can be used by the git client to communicate with\n> the remote end natively in a stateless-rpc manner as supported by\n> protocol v2. This means that the client must send everything the server\n> needs in a single request as the client must not assume any\n> state-storing on the part of the server or transport.\n>\n> If a stateless connection cannot be established then the remote-helper\n> will respond in the same manner as the 'connect' command indicating that\n> the client should fallback to using the dumb remote-helper commands.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> transport-helper.c | 8 ++++++++\n> transport.c | 1 +\n> transport.h | 6 ++++++\n> 3 files changed, 15 insertions(+)\n\nPlease add documentation for this command to\nDocumentation/gitremote-helpers.txt.\n\nThat helps reviewers, since it means reviewers can get a sense of what\nthe interface is meant to be. It helps remote helper implementers as\nwell: it tells them what they can rely on and what can't rely on in\nthis interface. For the same reason it helpers remote helper callers\nas well.\n\nThanks,\nJonathan\n"},{"id":"340488","messageId":"xmqq606inh4b.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180227221808.GE11187@sigill.intra.peff.net","subject":"Re: [PATCH v3 14/35] connect: request remote refs using v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-27T23:32:36Z","receivedAt":"2018-02-27T23:32:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> struct strs {...};\n>> \n>> void strs_init(struct strs *);\n>> void strs_push(struct strs *, const char *);\n>> void strs_pushf(struct strs *, const char *fmt, ...);\n>> void strs_pushl(struct strs *, ...);\n>> void strs_pushv(struct strs *, const char **);\n>> void strs_pop(struct strs *);\n>> void strs_clear(struct strs *);\n>> const char **strs_detach(struct strs *);\n>> \n>> ...is short, feels pretty natural, and doesn't require understanding\n>> \"v\" for \"vector\".\n>\n> Not bad. The \"v\" carries the information that it _is_ a NULL-terminated\n> vector and not some other list-like structure (and so is suitable for\n> feeding to execv, etc). But that may just be obvious from looking at its\n> uses and documentation.\n\nAnd with \"v\", it probably is obvious without looking at its uses and\ndocumentation, so... ;-)\n"},{"id":"340493","messageId":"20180228000520.GI174036@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180207011312.189834-35-bmwill@google.com","subject":"Re: [PATCH v3 34/35] remote-curl: implement stateless-connect command","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-02-28T00:05:20Z","receivedAt":"2018-02-28T00:05:34Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> Teach remote-curl the 'stateless-connect' command which is used to\n> establish a stateless connection with servers which support protocol\n> version 2. This allows remote-curl to act as a proxy, allowing the git\n> client to communicate natively with a remote end, simply using\n> remote-curl as a pass through to convert requests to http.\n\nCool! I better look at the spec for that first.\n\n*looks at the previous patch*\n\nOh, there is no documented spec. :/ I'll muddle through this instead,\nthen.\n\n[...]\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -188,7 +188,10 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n> \theads->version = discover_version(&reader);\n> \tswitch (heads->version) {\n> \tcase protocol_v2:\n> -\t\tdie(\"support for protocol v2 not implemented yet\");\n> +\t\t/*\n> +\t\t * Do nothing. Client should run 'stateless-connect' and\n> +\t\t * request the refs themselves.\n> +\t\t */\n> \t\tbreak;\n\nThis is the 'list' command, right? Since we expect the client to run\n'stateless-connect' instead, can we make it error out?\n\n[...]\n> @@ -1082,6 +1085,184 @@ static void parse_push(struct strbuf *buf)\n> \tfree(specs);\n> }\n> \n> +struct proxy_state {\n> +\tchar *service_name;\n> +\tchar *service_url;\n> +\tstruct curl_slist *headers;\n> +\tstruct strbuf request_buffer;\n> +\tint in;\n> +\tint out;\n> +\tstruct packet_reader reader;\n> +\tsize_t pos;\n> +\tint seen_flush;\n> +};\n\nCan this have a comment describing what it is/does? It's not obvious\nto me at first glance.\n\nIt doesn't have to go in a lot of detail since this is an internal\nimplementation detail, but something saying e.g. that this represents\na connection to an HTTP server (that's an example; I'm not saying\nthat's what it represents :)) would help.\n\n> +\n> +static void proxy_state_init(struct proxy_state *p, const char *service_name,\n> +\t\t\t enum protocol_version version)\n[...]\n> +static void proxy_state_clear(struct proxy_state *p)\n\nLooks sensible.\n\n[...]\n> +static size_t proxy_in(char *buffer, size_t eltsize,\n> +\t\t size_t nmemb, void *userdata)\n\nCan this have a comment describing the interface? (E.g. does it read\na single pkt_line? How is the caller expected to use it? Does it\nsatisfy the interface of some callback?)\n\nlibcurl's example https://curl.haxx.se/libcurl/c/ftpupload.html just\ncalls this read_callback. Such a name plus a pointer to\nCURLOPT_READFUNCTION should do the trick; bonus points if the comment \nsays what our implementation of the callback does.\n\nIs this about having peek ability?\n\n> +{\n> +\tsize_t max = eltsize * nmemb;\n\nCan this overflow? st_mult can avoid having to worry about that.\n\n> +\tstruct proxy_state *p = userdata;\n> +\tsize_t avail = p->request_buffer.len - p->pos;\n> +\n> +\tif (!avail) {\n> +\t\tif (p->seen_flush) {\n> +\t\t\tp->seen_flush = 0;\n> +\t\t\treturn 0;\n> +\t\t}\n> +\n> +\t\tstrbuf_reset(&p->request_buffer);\n> +\t\tswitch (packet_reader_read(&p->reader)) {\n> +\t\tcase PACKET_READ_EOF:\n> +\t\t\tdie(\"unexpected EOF when reading from parent process\");\n> +\t\tcase PACKET_READ_NORMAL:\n> +\t\t\tpacket_buf_write_len(&p->request_buffer, p->reader.line,\n> +\t\t\t\t\t p->reader.pktlen);\n> +\t\t\tbreak;\n> +\t\tcase PACKET_READ_DELIM:\n> +\t\t\tpacket_buf_delim(&p->request_buffer);\n> +\t\t\tbreak;\n> +\t\tcase PACKET_READ_FLUSH:\n> +\t\t\tpacket_buf_flush(&p->request_buffer);\n> +\t\t\tp->seen_flush = 1;\n> +\t\t\tbreak;\n> +\t\t}\n> +\t\tp->pos = 0;\n> +\t\tavail = p->request_buffer.len;\n> +\t}\n> +\n> +\tif (max < avail)\n> +\t\tavail = max;\n> +\tmemcpy(buffer, p->request_buffer.buf + p->pos, avail);\n> +\tp->pos += avail;\n> +\treturn avail;\n\nThis is a number of bytes, but CURLOPT_READFUNCTION expects a number\nof items, fread-style. That is:\n\n\tif (avail < eltsize)\n\t\t... handle somehow, maybe by reading in more? ...\n\n\tavail_memb = avail / eltsize;\n\tmemcpy(buffer,\n\t p->request_buffer.buf + p->pos,\n\t st_mult(avail_memb, eltsize));\n\tp->pos += st_mult(avail_memb, eltsize);\n\treturn avail_memb;\n\nBut https://curl.haxx.se/libcurl/c/CURLOPT_READFUNCTION.html says\n\n\tYour function must then return the actual number of bytes that\n\tit stored in that memory area.\n\nDoes this mean eltsize is always 1? This is super confusing...\n\n... ok, a quick grep for fread_func in libcurl reveals that eltsize is\nindeed always 1. Can we add an assertion so we notice if that\nchanges?\n\n\tif (eltsize != 1)\n\t\tBUG(\"curl read callback called with size = %zu != 1\", eltsize);\n\tmax = nmemb;\n\n[...]\n> +static size_t proxy_out(char *buffer, size_t eltsize,\n> +\t\t\tsize_t nmemb, void *userdata)\n> +{\n> +\tsize_t size = eltsize * nmemb;\n> +\tstruct proxy_state *p = userdata;\n> +\n> +\twrite_or_die(p->out, buffer, size);\n> +\treturn size;\n> +}\n\nNice. Same questions about st_mult or just asserting on eltsize apply\nhere, too.\n\n[...]\n> +static int proxy_post(struct proxy_state *p)\n\nWhat does this function do? Can it get a brief comment?\n\n> +{\n> +\tstruct active_request_slot *slot;\n> +\tint err;\n> +\n> +\tslot = get_active_slot();\n> +\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, p->headers);\n> +\n> +\t/* Setup function to read request from client */\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_READDATA, p);\n> +\n> +\t/* Setup function to write server response to client */\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);\n> +\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);\n> +\n> +\terr = run_slot(slot, NULL);\n> +\n> +\tif (err != HTTP_OK)\n> +\t\terr = -1;\n> +\n> +\treturn err;\n\nHTTP_OK is 0 but kind of obscures that. How about something like the\nfollowing?\n\n\tif (run_slot(slot, NULL))\n\t\treturn -1;\n\treturn 0;\n\nor\n\n\tif (run_slot(slot, NULL) != HTTP_OK)\n\t\treturn -1;\n\treturn 0;\n\nThat way it's clearer that this always returns 0 or -1.\n\n[...]\n> +static int stateless_connect(const char *service_name)\n> +{\n> +\tstruct discovery *discover;\n> +\tstruct proxy_state p;\n> +\n> +\t/*\n> +\t * Run the info/refs request and see if the server supports protocol\n> +\t * v2. If and only if the server supports v2 can we successfully\n> +\t * establish a stateless connection, otherwise we need to tell the\n> +\t * client to fallback to using other transport helper functions to\n> +\t * complete their request.\n> +\t */\n> +\tdiscover = discover_refs(service_name, 0);\n> +\tif (discover->version != protocol_v2) {\n> +\t\tprintf(\"fallback\\n\");\n> +\t\tfflush(stdout);\n> +\t\treturn -1;\n\nInteresting. I wonder if we can make remote-curl less smart and drive\nthis more from the caller.\n\nE.g. if the caller could do a single stateless request, they could do:\n\n\toption git-protocol version=2\n\tstateless-request GET info/refs?service=git-upload-pack\n\t[pkt-lines, ending with a flush-pkt]\n\nThe git-protocol option in this hypothetical example is the value to\nbe passed in the Git-Protocol header.\n\nThen based on the response, the caller could decide to keep using\nstateless-request for further requests or fall back to \"fetch\".\n\nThat way, if we implement some protocol v3, the remote helper would\nnot have to be changed at all to handle it: the caller would instead\nmake the new v3-format request and remote-curl would be able to oblige\nwithout knowing why they're doing it.\n\nWhat do you think?\n\nThanks,\nJonathan\n"},{"id":"340516","messageId":"20180228010821.GB45342@google.com","threadId":"47524","inReplyTo":"20180227231145.GF174036@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 29/35] pkt-line: add packet_buf_write_len function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T01:08:21Z","receivedAt":"2018-02-28T01:08:29Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/27, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> \n> > Add the 'packet_buf_write_len()' function which allows for writing an\n> > arbitrary length buffer into a 'struct strbuf' and formatting it in\n> > packet-line format.\n> \n> Makes sense.\n> \n> [...]\n> > --- a/pkt-line.h\n> > +++ b/pkt-line.h\n> > @@ -26,6 +26,7 @@ void packet_buf_flush(struct strbuf *buf);\n> > void packet_buf_delim(struct strbuf *buf);\n> > void packet_write(int fd_out, const char *buf, size_t size);\n> > void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n> > +void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len);\n> \n> I wonder if we should rename packet_buf_write to something like\n> packet_buf_writef. Right now there's a kind of confusing collection of\n> functions without much symmetry.\n> \n> Alternatively, the _buf_ ones could become strbuf_* functions:\n> \n> \tstrbuf_add_packet(&buf, data, len);\n> \tstrbuf_addf_packet(&buf, fmt, ...);\n> \n> That would make it clearer that these append to buf.\n> \n> I'm just thinking out loud. For this series, the API you have here\n> looks fine, even if it is a bit inconsistent. (In other words, even\n> if you agree with me, this would probably be best addressed as a patch\n> on top.)\n\nYeah I agree that an api change is needed, but yeah it can be done on\ntop of this series.\n\n> \n> [...]\n> > --- a/pkt-line.c\n> > +++ b/pkt-line.c\n> > @@ -215,6 +215,22 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n> > \tva_end(args);\n> > }\n> > \n> > +void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len)\n> > +{\n> > +\tsize_t orig_len, n;\n> > +\n> > +\torig_len = buf->len;\n> > +\tstrbuf_addstr(buf, \"0000\");\n> > +\tstrbuf_add(buf, data, len);\n> > +\tn = buf->len - orig_len;\n> > +\n> > +\tif (n > LARGE_PACKET_MAX)\n> > +\t\tdie(\"protocol error: impossibly long line\");\n> \n> Could the error message describe the long line (e.g.\n> \n> \t\t...impossibly long line %.*s...\", 256, data);\n> \n\nI was reusing the error msg as it appears in another part of this file.\n\n> )?\n> \n> > +\n> > +\tset_packet_header(&buf->buf[orig_len], n);\n> > +\tpacket_trace(buf->buf + orig_len + 4, n - 4, 1);\n> \n> Could do, more simply:\n> \n> \tpacket_trace(data, len, 1);\n\nI'll change this.\n\n> \n> Thanks,\n> Jonathan\n\n-- \nBrandon Williams\n"},{"id":"340604","messageId":"20180228190917.GB87122@google.com","threadId":"47524","inReplyTo":"20180227233025.GH174036@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 28/35] transport-helper: introduce stateless-connect","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T19:09:17Z","receivedAt":"2018-02-28T19:09:25Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/27, Jonathan Nieder wrote:\n> Brandon Williams wrote:\n> \n> > Introduce the transport-helper capability 'stateless-connect'. This\n> > capability indicates that the transport-helper can be requested to run\n> > the 'stateless-connect' command which should attempt to make a\n> > stateless connection with a remote end. Once established, the\n> > connection can be used by the git client to communicate with\n> > the remote end natively in a stateless-rpc manner as supported by\n> > protocol v2. This means that the client must send everything the server\n> > needs in a single request as the client must not assume any\n> > state-storing on the part of the server or transport.\n> >\n> > If a stateless connection cannot be established then the remote-helper\n> > will respond in the same manner as the 'connect' command indicating that\n> > the client should fallback to using the dumb remote-helper commands.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > transport-helper.c | 8 ++++++++\n> > transport.c | 1 +\n> > transport.h | 6 ++++++\n> > 3 files changed, 15 insertions(+)\n> \n> Please add documentation for this command to\n> Documentation/gitremote-helpers.txt.\n> \n> That helps reviewers, since it means reviewers can get a sense of what\n> the interface is meant to be. It helps remote helper implementers as\n> well: it tells them what they can rely on and what can't rely on in\n> this interface. For the same reason it helpers remote helper callers\n> as well.\n> \n> Thanks,\n> Jonathan\n\nThanks for reminding me. I had intended to at some point but had\nforgotten to do so. I'm going to mark this it as experimental and for\ninternal use only so that we can still tweak the interface if we want\nbefore it becomes stable.\n\n-- \nBrandon Williams\n"},{"id":"340612","messageId":"20180228202142.GC87122@google.com","threadId":"47524","inReplyTo":"20180228000520.GI174036@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 34/35] remote-curl: implement stateless-connect command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T20:21:42Z","receivedAt":"2018-02-28T20:21:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 02/27, Jonathan Nieder wrote:\n> Hi,\n> \n> Brandon Williams wrote:\n> \n> > Teach remote-curl the 'stateless-connect' command which is used to\n> > establish a stateless connection with servers which support protocol\n> > version 2. This allows remote-curl to act as a proxy, allowing the git\n> > client to communicate natively with a remote end, simply using\n> > remote-curl as a pass through to convert requests to http.\n> \n> Cool! I better look at the spec for that first.\n> \n> *looks at the previous patch*\n> \n> Oh, there is no documented spec. :/ I'll muddle through this instead,\n> then.\n\nI'll make sure to add one :)\n\n> \n> [...]\n> > --- a/remote-curl.c\n> > +++ b/remote-curl.c\n> > @@ -188,7 +188,10 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n> > \theads->version = discover_version(&reader);\n> > \tswitch (heads->version) {\n> > \tcase protocol_v2:\n> > -\t\tdie(\"support for protocol v2 not implemented yet\");\n> > +\t\t/*\n> > +\t\t * Do nothing. Client should run 'stateless-connect' and\n> > +\t\t * request the refs themselves.\n> > +\t\t */\n> > \t\tbreak;\n> \n> This is the 'list' command, right? Since we expect the client to run\n> 'stateless-connect' instead, can we make it error out?\n\nYes and no. Remote-curl will run this when trying to establish a\nstateless-connection. If the response is v2 then this is a capability\nlist and not refs. So the capabilities will be dumped to the client and\nthey will be able to request the refs themselves at a later point. The\ncomment here is just misleading, so i'll make sure to fix it.\n\n> \n> [...]\n> > @@ -1082,6 +1085,184 @@ static void parse_push(struct strbuf *buf)\n> > \tfree(specs);\n> > }\n> > \n> > +struct proxy_state {\n> > +\tchar *service_name;\n> > +\tchar *service_url;\n> > +\tstruct curl_slist *headers;\n> > +\tstruct strbuf request_buffer;\n> > +\tint in;\n> > +\tint out;\n> > +\tstruct packet_reader reader;\n> > +\tsize_t pos;\n> > +\tint seen_flush;\n> > +};\n> \n> Can this have a comment describing what it is/does? It's not obvious\n> to me at first glance.\n> \n> It doesn't have to go in a lot of detail since this is an internal\n> implementation detail, but something saying e.g. that this represents\n> a connection to an HTTP server (that's an example; I'm not saying\n> that's what it represents :)) would help.\n\nAlways making new code have higher standards than the existing code ;)\nHaha, I'll add a simple comment explaining it.\n\n> \n> > +\n> > +static void proxy_state_init(struct proxy_state *p, const char *service_name,\n> > +\t\t\t enum protocol_version version)\n> [...]\n> > +static void proxy_state_clear(struct proxy_state *p)\n> \n> Looks sensible.\n> \n> [...]\n> > +static size_t proxy_in(char *buffer, size_t eltsize,\n> > +\t\t size_t nmemb, void *userdata)\n> \n> Can this have a comment describing the interface? (E.g. does it read\n> a single pkt_line? How is the caller expected to use it? Does it\n> satisfy the interface of some callback?)\n\nI'll add a comment that its used as a READFUNCTION callback for curl and\nthat it tries to copy over a packet-line at a time.\n\n> \n> libcurl's example https://curl.haxx.se/libcurl/c/ftpupload.html just\n> calls this read_callback. Such a name plus a pointer to\n> CURLOPT_READFUNCTION should do the trick; bonus points if the comment \n> says what our implementation of the callback does.\n> \n> Is this about having peek ability?\n\nNo its just that Curl only requests a set about of data at a time so you\nneed to be able to buffer the data that can't be read yet.\n\n> \n> > +\tstruct proxy_state *p = userdata;\n> > +\tsize_t avail = p->request_buffer.len - p->pos;\n> > +\n> > +\tif (!avail) {\n> > +\t\tif (p->seen_flush) {\n> > +\t\t\tp->seen_flush = 0;\n> > +\t\t\treturn 0;\n> > +\t\t}\n> > +\n> > +\t\tstrbuf_reset(&p->request_buffer);\n> > +\t\tswitch (packet_reader_read(&p->reader)) {\n> > +\t\tcase PACKET_READ_EOF:\n> > +\t\t\tdie(\"unexpected EOF when reading from parent process\");\n> > +\t\tcase PACKET_READ_NORMAL:\n> > +\t\t\tpacket_buf_write_len(&p->request_buffer, p->reader.line,\n> > +\t\t\t\t\t p->reader.pktlen);\n> > +\t\t\tbreak;\n> > +\t\tcase PACKET_READ_DELIM:\n> > +\t\t\tpacket_buf_delim(&p->request_buffer);\n> > +\t\t\tbreak;\n> > +\t\tcase PACKET_READ_FLUSH:\n> > +\t\t\tpacket_buf_flush(&p->request_buffer);\n> > +\t\t\tp->seen_flush = 1;\n> > +\t\t\tbreak;\n> > +\t\t}\n> > +\t\tp->pos = 0;\n> > +\t\tavail = p->request_buffer.len;\n> > +\t}\n> > +\n> > +\tif (max < avail)\n> > +\t\tavail = max;\n> > +\tmemcpy(buffer, p->request_buffer.buf + p->pos, avail);\n> > +\tp->pos += avail;\n> > +\treturn avail;\n> \n> This is a number of bytes, but CURLOPT_READFUNCTION expects a number\n> of items, fread-style. That is:\n> \n> \tif (avail < eltsize)\n> \t\t... handle somehow, maybe by reading in more? ...\n> \n> \tavail_memb = avail / eltsize;\n> \tmemcpy(buffer,\n> \t p->request_buffer.buf + p->pos,\n> \t st_mult(avail_memb, eltsize));\n> \tp->pos += st_mult(avail_memb, eltsize);\n> \treturn avail_memb;\n> \n> But https://curl.haxx.se/libcurl/c/CURLOPT_READFUNCTION.html says\n> \n> \tYour function must then return the actual number of bytes that\n> \tit stored in that memory area.\n> \n> Does this mean eltsize is always 1? This is super confusing...\n> \n> ... ok, a quick grep for fread_func in libcurl reveals that eltsize is\n> indeed always 1. Can we add an assertion so we notice if that\n> changes?\n> \n> \tif (eltsize != 1)\n> \t\tBUG(\"curl read callback called with size = %zu != 1\", eltsize);\n> \tmax = nmemb;\n\nYeah i can go ahead and do this. Just note that the v1 path uses logic\nidentical to this so it would be a problem there.\n\n> \n> [...]\n> > +static size_t proxy_out(char *buffer, size_t eltsize,\n> > +\t\t\tsize_t nmemb, void *userdata)\n> > +{\n> > +\tsize_t size = eltsize * nmemb;\n> > +\tstruct proxy_state *p = userdata;\n> > +\n> > +\twrite_or_die(p->out, buffer, size);\n> > +\treturn size;\n> > +}\n> \n> Nice. Same questions about st_mult or just asserting on eltsize apply\n> here, too.\n> \n> [...]\n> > +static int proxy_post(struct proxy_state *p)\n> \n> What does this function do? Can it get a brief comment?\n\nWill do.\n\n> \n> > +{\n> > +\tstruct active_request_slot *slot;\n> > +\tint err;\n> > +\n> > +\tslot = get_active_slot();\n> > +\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, p->headers);\n> > +\n> > +\t/* Setup function to read request from client */\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_READDATA, p);\n> > +\n> > +\t/* Setup function to write server response to client */\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);\n> > +\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);\n> > +\n> > +\terr = run_slot(slot, NULL);\n> > +\n> > +\tif (err != HTTP_OK)\n> > +\t\terr = -1;\n> > +\n> > +\treturn err;\n> \n> HTTP_OK is 0 but kind of obscures that. How about something like the\n> following?\n> \n> \tif (run_slot(slot, NULL))\n> \t\treturn -1;\n> \treturn 0;\n> \n> or\n> \n> \tif (run_slot(slot, NULL) != HTTP_OK)\n> \t\treturn -1;\n> \treturn 0;\n> \n> That way it's clearer that this always returns 0 or -1.\n\nSounds good.\n\n> \n> [...]\n> > +static int stateless_connect(const char *service_name)\n> > +{\n> > +\tstruct discovery *discover;\n> > +\tstruct proxy_state p;\n> > +\n> > +\t/*\n> > +\t * Run the info/refs request and see if the server supports protocol\n> > +\t * v2. If and only if the server supports v2 can we successfully\n> > +\t * establish a stateless connection, otherwise we need to tell the\n> > +\t * client to fallback to using other transport helper functions to\n> > +\t * complete their request.\n> > +\t */\n> > +\tdiscover = discover_refs(service_name, 0);\n> > +\tif (discover->version != protocol_v2) {\n> > +\t\tprintf(\"fallback\\n\");\n> > +\t\tfflush(stdout);\n> > +\t\treturn -1;\n> \n> Interesting. I wonder if we can make remote-curl less smart and drive\n> this more from the caller.\n> \n> E.g. if the caller could do a single stateless request, they could do:\n> \n> \toption git-protocol version=2\n> \tstateless-request GET info/refs?service=git-upload-pack\n> \t[pkt-lines, ending with a flush-pkt]\n> \n> The git-protocol option in this hypothetical example is the value to\n> be passed in the Git-Protocol header.\n> \n> Then based on the response, the caller could decide to keep using\n> stateless-request for further requests or fall back to \"fetch\".\n> \n> That way, if we implement some protocol v3, the remote helper would\n> not have to be changed at all to handle it: the caller would instead\n> make the new v3-format request and remote-curl would be able to oblige\n> without knowing why they're doing it.\n> \n> What do you think?\n\nI do see the draw for wanting this. I think a change like this requires\na lot more refactoring, simply because with the current setup the\nfetch/ls-refs logic doesn't care that its talking through a\nremote-helper where if we went down that route it would need to be aware\nof that.\n\n-- \nBrandon Williams\n"},{"id":"340641","messageId":"20180228232252.102167-1-bmwill@google.com","threadId":"47524","inReplyTo":"20180207011312.189834-1-bmwill@google.com","subject":"[PATCH v4 00/35] protocol version 2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:17Z","receivedAt":"2018-02-28T23:23:23Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Lots of changes since v3 (well more than between v2 and v3). Thanks for\nall of the reviews on the last round, the series is getting more\npolished.\n\n * Eliminated the \"# service\" line from the response from an HTTP\n server. This means that the response to a v2 request is exactly the\n same regardless of which transport you use! Docs for this have been\n added as well.\n * Changed how ref-patterns work with the `ls-refs` command. Instead of\n using wildmatch all patterns must either match exactly or they can\n contain a single '*' character at the end to mean that the prefix\n must match. Docs for this have also been added.\n * Lots of updates to the docs. Including documenting the\n `stateless-connect` remote-helper command used by remote-curl to\n handle the http transport.\n * Fixed a number of bugs with the `fetch` command, one of which didn't\n use objects from configured alternates.\n\nBrandon Williams (35):\n pkt-line: introduce packet_read_with_status\n pkt-line: allow peeking a packet line without consuming it\n pkt-line: add delim packet support\n upload-pack: convert to a builtin\n upload-pack: factor out processing lines\n transport: use get_refs_via_connect to get refs\n connect: convert get_remote_heads to use struct packet_reader\n connect: discover protocol version outside of get_remote_heads\n transport: store protocol version\n protocol: introduce enum protocol_version value protocol_v2\n test-pkt-line: introduce a packet-line test helper\n serve: introduce git-serve\n ls-refs: introduce ls-refs server command\n connect: request remote refs using v2\n transport: convert get_refs_list to take a list of ref patterns\n transport: convert transport_get_remote_refs to take a list of ref\n patterns\n ls-remote: pass ref patterns when requesting a remote's refs\n fetch: pass ref patterns when fetching\n push: pass ref patterns when pushing\n upload-pack: introduce fetch server command\n fetch-pack: perform a fetch using v2\n fetch-pack: support shallow requests\n connect: refactor git_connect to only get the protocol version once\n connect: don't request v2 when pushing\n transport-helper: remove name parameter\n transport-helper: refactor process_connect_service\n transport-helper: introduce stateless-connect\n pkt-line: add packet_buf_write_len function\n remote-curl: create copy of the service name\n remote-curl: store the protocol version the server responded with\n http: allow providing extra headers for http requests\n http: don't always add Git-Protocol header\n http: eliminate \"# service\" line when using protocol v2\n remote-curl: implement stateless-connect command\n remote-curl: don't request v2 when pushing\n\n .gitignore | 1 +\n Documentation/gitremote-helpers.txt | 32 ++\n Documentation/technical/protocol-v2.txt | 401 +++++++++++++++\n Makefile | 7 +-\n builtin.h | 2 +\n builtin/clone.c | 2 +-\n builtin/fetch-pack.c | 20 +-\n builtin/fetch.c | 18 +-\n builtin/ls-remote.c | 12 +-\n builtin/receive-pack.c | 6 +\n builtin/remote.c | 2 +-\n builtin/send-pack.c | 20 +-\n builtin/serve.c | 30 ++\n builtin/upload-pack.c | 74 +++\n connect.c | 364 ++++++++++----\n connect.h | 7 +\n fetch-pack.c | 339 ++++++++++++-\n fetch-pack.h | 4 +-\n git.c | 2 +\n http-backend.c | 8 +-\n http.c | 25 +-\n http.h | 7 +\n ls-refs.c | 144 ++++++\n ls-refs.h | 9 +\n pkt-line.c | 147 +++++-\n pkt-line.h | 78 +++\n protocol.c | 2 +\n protocol.h | 1 +\n refs.c | 14 +\n refs.h | 7 +\n remote-curl.c | 278 ++++++++++-\n remote.h | 11 +-\n serve.c | 260 ++++++++++\n serve.h | 15 +\n t/helper/test-pkt-line.c | 64 +++\n t/t5701-git-serve.sh | 176 +++++++\n t/t5702-protocol-v2.sh | 273 +++++++++++\n transport-helper.c | 87 ++--\n transport-internal.h | 9 +-\n transport.c | 125 +++--\n transport.h | 18 +-\n upload-pack.c | 616 ++++++++++++++++++------\n upload-pack.h | 21 +\n 43 files changed, 3370 insertions(+), 368 deletions(-)\n create mode 100644 Documentation/technical/protocol-v2.txt\n create mode 100644 builtin/serve.c\n create mode 100644 builtin/upload-pack.c\n create mode 100644 ls-refs.c\n create mode 100644 ls-refs.h\n create mode 100644 serve.c\n create mode 100644 serve.h\n create mode 100644 t/helper/test-pkt-line.c\n create mode 100755 t/t5701-git-serve.sh\n create mode 100755 t/t5702-protocol-v2.sh\n create mode 100644 upload-pack.h\n\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340642","messageId":"20180228232252.102167-2-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 01/35] pkt-line: introduce packet_read_with_status","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:18Z","receivedAt":"2018-02-28T23:23:25Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"The current pkt-line API encodes the status of a pkt-line read in the\nlength of the read content. An error is indicated with '-1', a flush\nwith '0' (which can be confusing since a return value of '0' can also\nindicate an empty pkt-line), and a positive integer for the length of\nthe read content otherwise. This doesn't leave much room for allowing\nthe addition of additional special packets in the future.\n\nTo solve this introduce 'packet_read_with_status()' which reads a packet\nand returns the status of the read encoded as an 'enum packet_status'\ntype. This allows for easily identifying between special and normal\npackets as well as errors. It also enables easily adding a new special\npacket in the future.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 55 ++++++++++++++++++++++++++++++++++++++++--------------\n pkt-line.h | 16 ++++++++++++++++\n 2 files changed, 57 insertions(+), 14 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 2827ca772..08e5ba44c 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -280,28 +280,34 @@ static int packet_length(const char *linelen)\n \treturn (val < 0) ? val : (val << 8) | hex2chr(linelen + 2);\n }\n \n-int packet_read(int fd, char **src_buf, size_t *src_len,\n-\t\tchar *buffer, unsigned size, int options)\n+enum packet_read_status packet_read_with_status(int fd, char **src_buffer,\n+\t\t\t\t\t\tsize_t *src_len, char *buffer,\n+\t\t\t\t\t\tunsigned size, int *pktlen,\n+\t\t\t\t\t\tint options)\n {\n-\tint len, ret;\n+\tint len;\n \tchar linelen[4];\n \n-\tret = get_packet_data(fd, src_buf, src_len, linelen, 4, options);\n-\tif (ret < 0)\n-\t\treturn ret;\n+\tif (get_packet_data(fd, src_buffer, src_len, linelen, 4, options) < 0)\n+\t\treturn PACKET_READ_EOF;\n+\n \tlen = packet_length(linelen);\n-\tif (len < 0)\n+\n+\tif (len < 0) {\n \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n-\tif (!len) {\n+\t} else if (!len) {\n \t\tpacket_trace(\"0000\", 4, 0);\n-\t\treturn 0;\n+\t\treturn PACKET_READ_FLUSH;\n+\t} else if (len < 4) {\n+\t\tdie(\"protocol error: bad line length %d\", len);\n \t}\n+\n \tlen -= 4;\n-\tif (len >= size)\n+\tif ((unsigned)len >= size)\n \t\tdie(\"protocol error: bad line length %d\", len);\n-\tret = get_packet_data(fd, src_buf, src_len, buffer, len, options);\n-\tif (ret < 0)\n-\t\treturn ret;\n+\n+\tif (get_packet_data(fd, src_buffer, src_len, buffer, len, options) < 0)\n+\t\treturn PACKET_READ_EOF;\n \n \tif ((options & PACKET_READ_CHOMP_NEWLINE) &&\n \t len && buffer[len-1] == '\\n')\n@@ -309,7 +315,28 @@ int packet_read(int fd, char **src_buf, size_t *src_len,\n \n \tbuffer[len] = 0;\n \tpacket_trace(buffer, len, 0);\n-\treturn len;\n+\t*pktlen = len;\n+\treturn PACKET_READ_NORMAL;\n+}\n+\n+int packet_read(int fd, char **src_buffer, size_t *src_len,\n+\t\tchar *buffer, unsigned size, int options)\n+{\n+\tint pktlen;\n+\n+\tswitch (packet_read_with_status(fd, src_buffer, src_len, buffer, size,\n+\t\t\t\t\t&pktlen, options)) {\n+\tcase PACKET_READ_EOF:\n+\t\tpktlen = -1;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\tbreak;\n+\tcase PACKET_READ_FLUSH:\n+\t\tpktlen = 0;\n+\t\tbreak;\n+\t}\n+\n+\treturn pktlen;\n }\n \n static char *packet_read_line_generic(int fd,\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3dad583e2..0be691116 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -65,6 +65,22 @@ int write_packetized_from_buf(const char *src_in, size_t len, int fd_out);\n int packet_read(int fd, char **src_buffer, size_t *src_len, char\n \t\t*buffer, unsigned size, int options);\n \n+/*\n+ * Read a packetized line into a buffer like the 'packet_read()' function but\n+ * returns an 'enum packet_read_status' which indicates the status of the read.\n+ * The number of bytes read will be assigined to *pktlen if the status of the\n+ * read was 'PACKET_READ_NORMAL'.\n+ */\n+enum packet_read_status {\n+\tPACKET_READ_EOF = -1,\n+\tPACKET_READ_NORMAL,\n+\tPACKET_READ_FLUSH,\n+};\n+enum packet_read_status packet_read_with_status(int fd, char **src_buffer,\n+\t\t\t\t\t\tsize_t *src_len, char *buffer,\n+\t\t\t\t\t\tunsigned size, int *pktlen,\n+\t\t\t\t\t\tint options);\n+\n /*\n * Convenience wrapper for packet_read that is not gentle, and sets the\n * CHOMP_NEWLINE option. The return value is NULL for a flush packet,\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340643","messageId":"20180228232252.102167-3-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 02/35] pkt-line: allow peeking a packet line without consuming it","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:19Z","receivedAt":"2018-02-28T23:23:28Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Sometimes it is advantageous to be able to peek the next packet line\nwithout consuming it (e.g. to be able to determine the protocol version\na server is speaking). In order to do that introduce 'struct\npacket_reader' which is an abstraction around the normal packet reading\nlogic. This enables a caller to be able to peek a single line at a time\nusing 'packet_reader_peek()' and having a caller consume a line by\ncalling 'packet_reader_read()'.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n pkt-line.h | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 117 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 08e5ba44c..6307fa4a3 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -404,3 +404,62 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out)\n \t}\n \treturn sb_out->len - orig_len;\n }\n+\n+/* Packet Reader Functions */\n+void packet_reader_init(struct packet_reader *reader, int fd,\n+\t\t\tchar *src_buffer, size_t src_len,\n+\t\t\tint options)\n+{\n+\tmemset(reader, 0, sizeof(*reader));\n+\n+\treader->fd = fd;\n+\treader->src_buffer = src_buffer;\n+\treader->src_len = src_len;\n+\treader->buffer = packet_buffer;\n+\treader->buffer_size = sizeof(packet_buffer);\n+\treader->options = options;\n+}\n+\n+enum packet_read_status packet_reader_read(struct packet_reader *reader)\n+{\n+\tif (reader->line_peeked) {\n+\t\treader->line_peeked = 0;\n+\t\treturn reader->status;\n+\t}\n+\n+\treader->status = packet_read_with_status(reader->fd,\n+\t\t\t\t\t\t &reader->src_buffer,\n+\t\t\t\t\t\t &reader->src_len,\n+\t\t\t\t\t\t reader->buffer,\n+\t\t\t\t\t\t reader->buffer_size,\n+\t\t\t\t\t\t &reader->pktlen,\n+\t\t\t\t\t\t reader->options);\n+\n+\tswitch (reader->status) {\n+\tcase PACKET_READ_EOF:\n+\t\treader->pktlen = -1;\n+\t\treader->line = NULL;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\treader->line = reader->buffer;\n+\t\tbreak;\n+\tcase PACKET_READ_FLUSH:\n+\t\treader->pktlen = 0;\n+\t\treader->line = NULL;\n+\t\tbreak;\n+\t}\n+\n+\treturn reader->status;\n+}\n+\n+enum packet_read_status packet_reader_peek(struct packet_reader *reader)\n+{\n+\t/* Only allow peeking a single line */\n+\tif (reader->line_peeked)\n+\t\treturn reader->status;\n+\n+\t/* Peek a line by reading it and setting peeked flag */\n+\tpacket_reader_read(reader);\n+\treader->line_peeked = 1;\n+\treturn reader->status;\n+}\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 0be691116..f2edfae9a 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -112,6 +112,64 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size);\n */\n ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out);\n \n+struct packet_reader {\n+\t/* source file descriptor */\n+\tint fd;\n+\n+\t/* source buffer and its size */\n+\tchar *src_buffer;\n+\tsize_t src_len;\n+\n+\t/* buffer that pkt-lines are read into and its size */\n+\tchar *buffer;\n+\tunsigned buffer_size;\n+\n+\t/* options to be used during reads */\n+\tint options;\n+\n+\t/* status of the last read */\n+\tenum packet_read_status status;\n+\n+\t/* length of data read during the last read */\n+\tint pktlen;\n+\n+\t/* the last line read */\n+\tconst char *line;\n+\n+\t/* indicates if a line has been peeked */\n+\tint line_peeked;\n+};\n+\n+/*\n+ * Initialize a 'struct packet_reader' object which is an\n+ * abstraction around the 'packet_read_with_status()' function.\n+ */\n+extern void packet_reader_init(struct packet_reader *reader, int fd,\n+\t\t\t char *src_buffer, size_t src_len,\n+\t\t\t int options);\n+\n+/*\n+ * Perform a packet read and return the status of the read.\n+ * The values of 'pktlen' and 'line' are updated based on the status of the\n+ * read as follows:\n+ *\n+ * PACKET_READ_ERROR: 'pktlen' is set to '-1' and 'line' is set to NULL\n+ * PACKET_READ_NORMAL: 'pktlen' is set to the number of bytes read\n+ *\t\t 'line' is set to point at the read line\n+ * PACKET_READ_FLUSH: 'pktlen' is set to '0' and 'line' is set to NULL\n+ */\n+extern enum packet_read_status packet_reader_read(struct packet_reader *reader);\n+\n+/*\n+ * Peek the next packet line without consuming it and return the status.\n+ * The next call to 'packet_reader_read()' will perform a read of the same line\n+ * that was peeked, consuming the line.\n+ *\n+ * Peeking multiple times without calling 'packet_reader_read()' will return\n+ * the same result.\n+ */\n+extern enum packet_read_status packet_reader_peek(struct packet_reader *reader);\n+\n #define DEFAULT_PACKET_MAX 1000\n #define LARGE_PACKET_MAX 65520\n #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4)\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340644","messageId":"20180228232252.102167-5-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 04/35] upload-pack: convert to a builtin","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:21Z","receivedAt":"2018-02-28T23:23:33Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to allow for code sharing with the server-side of fetch in\nprotocol-v2 convert upload-pack to be a builtin.\n\nSigned-off-by: Brandon Williams \n---\n Makefile | 3 +-\n builtin.h | 1 +\n builtin/upload-pack.c | 67 ++++++++++++++++++++++++++\n git.c | 1 +\n upload-pack.c | 107 ++++++++++--------------------------------\n upload-pack.h | 13 +++++\n 6 files changed, 109 insertions(+), 83 deletions(-)\n create mode 100644 builtin/upload-pack.c\n create mode 100644 upload-pack.h\n\ndiff --git a/Makefile b/Makefile\nindex 1a9b23b67..b7ccc05fa 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -639,7 +639,6 @@ PROGRAM_OBJS += imap-send.o\n PROGRAM_OBJS += sh-i18n--envsubst.o\n PROGRAM_OBJS += shell.o\n PROGRAM_OBJS += show-index.o\n-PROGRAM_OBJS += upload-pack.o\n PROGRAM_OBJS += remote-testsvn.o\n \n # Binary suffix, set to .exe for Windows builds\n@@ -909,6 +908,7 @@ LIB_OBJS += tree-diff.o\n LIB_OBJS += tree.o\n LIB_OBJS += tree-walk.o\n LIB_OBJS += unpack-trees.o\n+LIB_OBJS += upload-pack.o\n LIB_OBJS += url.o\n LIB_OBJS += urlmatch.o\n LIB_OBJS += usage.o\n@@ -1026,6 +1026,7 @@ BUILTIN_OBJS += builtin/update-index.o\n BUILTIN_OBJS += builtin/update-ref.o\n BUILTIN_OBJS += builtin/update-server-info.o\n BUILTIN_OBJS += builtin/upload-archive.o\n+BUILTIN_OBJS += builtin/upload-pack.o\n BUILTIN_OBJS += builtin/var.o\n BUILTIN_OBJS += builtin/verify-commit.o\n BUILTIN_OBJS += builtin/verify-pack.o\ndiff --git a/builtin.h b/builtin.h\nindex 42378f3aa..f332a1257 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -231,6 +231,7 @@ extern int cmd_update_ref(int argc, const char **argv, const char *prefix);\n extern int cmd_update_server_info(int argc, const char **argv, const char *prefix);\n extern int cmd_upload_archive(int argc, const char **argv, const char *prefix);\n extern int cmd_upload_archive_writer(int argc, const char **argv, const char *prefix);\n+extern int cmd_upload_pack(int argc, const char **argv, const char *prefix);\n extern int cmd_var(int argc, const char **argv, const char *prefix);\n extern int cmd_verify_commit(int argc, const char **argv, const char *prefix);\n extern int cmd_verify_tag(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\nnew file mode 100644\nindex 000000000..2cb5cb35b\n--- /dev/null\n+++ b/builtin/upload-pack.c\n@@ -0,0 +1,67 @@\n+#include \"cache.h\"\n+#include \"builtin.h\"\n+#include \"exec_cmd.h\"\n+#include \"pkt-line.h\"\n+#include \"parse-options.h\"\n+#include \"protocol.h\"\n+#include \"upload-pack.h\"\n+\n+static const char * const upload_pack_usage[] = {\n+\tN_(\"git upload-pack [] \"),\n+\tNULL\n+};\n+\n+int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n+{\n+\tconst char *dir;\n+\tint strict = 0;\n+\tstruct upload_pack_options opts = { 0 };\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n+\t\t\t N_(\"quit after a single request/response exchange\")),\n+\t\tOPT_BOOL(0, \"advertise-refs\", &opts.advertise_refs,\n+\t\t\t N_(\"exit immediately after initial ref advertisement\")),\n+\t\tOPT_BOOL(0, \"strict\", &strict,\n+\t\t\t N_(\"do not try /.git/ if is no Git directory\")),\n+\t\tOPT_INTEGER(0, \"timeout\", &opts.timeout,\n+\t\t\t N_(\"interrupt transfer after seconds of inactivity\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tpacket_trace_identity(\"upload-pack\");\n+\tcheck_replace_refs = 0;\n+\n+\targc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);\n+\n+\tif (argc != 1)\n+\t\tusage_with_options(upload_pack_usage, options);\n+\n+\tif (opts.timeout)\n+\t\topts.daemon_mode = 1;\n+\n+\tsetup_path();\n+\n+\tdir = argv[0];\n+\n+\tif (!enter_repo(dir, strict))\n+\t\tdie(\"'%s' does not appear to be a git repository\", dir);\n+\n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\t\tif (opts.advertise_refs || !opts.stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\n+\t\t/* fallthrough */\n+\tcase protocol_v0:\n+\t\tupload_pack(&opts);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex c870b9719..f71073dc8 100644\n--- a/git.c\n+++ b/git.c\n@@ -478,6 +478,7 @@ static struct cmd_struct commands[] = {\n \t{ \"update-server-info\", cmd_update_server_info, RUN_SETUP },\n \t{ \"upload-archive\", cmd_upload_archive },\n \t{ \"upload-archive--writer\", cmd_upload_archive_writer },\n+\t{ \"upload-pack\", cmd_upload_pack },\n \t{ \"var\", cmd_var, RUN_SETUP_GENTLY },\n \t{ \"verify-commit\", cmd_verify_commit, RUN_SETUP },\n \t{ \"verify-pack\", cmd_verify_pack },\ndiff --git a/upload-pack.c b/upload-pack.c\nindex d5de18127..2ad73a98b 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -6,7 +6,6 @@\n #include \"tag.h\"\n #include \"object.h\"\n #include \"commit.h\"\n-#include \"exec_cmd.h\"\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n@@ -15,15 +14,10 @@\n #include \"sigchain.h\"\n #include \"version.h\"\n #include \"string-list.h\"\n-#include \"parse-options.h\"\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n #include \"protocol.h\"\n-\n-static const char * const upload_pack_usage[] = {\n-\tN_(\"git upload-pack [] \"),\n-\tNULL\n-};\n+#include \"upload-pack.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -61,7 +55,6 @@ static int keepalive = 5;\n * otherwise maximum packet size (up to 65520 bytes).\n */\n static int use_sideband;\n-static int advertise_refs;\n static int stateless_rpc;\n static const char *pack_objects_hook;\n \n@@ -977,33 +970,6 @@ static int find_symref(const char *refname, const struct object_id *oid,\n \treturn 0;\n }\n \n-static void upload_pack(void)\n-{\n-\tstruct string_list symref = STRING_LIST_INIT_DUP;\n-\n-\thead_ref_namespaced(find_symref, &symref);\n-\n-\tif (advertise_refs || !stateless_rpc) {\n-\t\treset_timeout();\n-\t\thead_ref_namespaced(send_ref, &symref);\n-\t\tfor_each_namespaced_ref(send_ref, &symref);\n-\t\tadvertise_shallow_grafts(1);\n-\t\tpacket_flush(1);\n-\t} else {\n-\t\thead_ref_namespaced(check_ref, NULL);\n-\t\tfor_each_namespaced_ref(check_ref, NULL);\n-\t}\n-\tstring_list_clear(&symref, 1);\n-\tif (advertise_refs)\n-\t\treturn;\n-\n-\treceive_needs();\n-\tif (want_obj.nr) {\n-\t\tget_common_commits();\n-\t\tcreate_pack_file();\n-\t}\n-}\n-\n static int upload_pack_config(const char *var, const char *value, void *unused)\n {\n \tif (!strcmp(\"uploadpack.allowtipsha1inwant\", var)) {\n@@ -1032,58 +998,35 @@ static int upload_pack_config(const char *var, const char *value, void *unused)\n \treturn parse_hide_refs_config(var, value, \"uploadpack\");\n }\n \n-int cmd_main(int argc, const char **argv)\n+void upload_pack(struct upload_pack_options *options)\n {\n-\tconst char *dir;\n-\tint strict = 0;\n-\tstruct option options[] = {\n-\t\tOPT_BOOL(0, \"stateless-rpc\", &stateless_rpc,\n-\t\t\t N_(\"quit after a single request/response exchange\")),\n-\t\tOPT_BOOL(0, \"advertise-refs\", &advertise_refs,\n-\t\t\t N_(\"exit immediately after initial ref advertisement\")),\n-\t\tOPT_BOOL(0, \"strict\", &strict,\n-\t\t\t N_(\"do not try /.git/ if is no Git directory\")),\n-\t\tOPT_INTEGER(0, \"timeout\", &timeout,\n-\t\t\t N_(\"interrupt transfer after seconds of inactivity\")),\n-\t\tOPT_END()\n-\t};\n-\n-\tpacket_trace_identity(\"upload-pack\");\n-\tcheck_replace_refs = 0;\n-\n-\targc = parse_options(argc, argv, NULL, options, upload_pack_usage, 0);\n-\n-\tif (argc != 1)\n-\t\tusage_with_options(upload_pack_usage, options);\n-\n-\tif (timeout)\n-\t\tdaemon_mode = 1;\n-\n-\tsetup_path();\n-\n-\tdir = argv[0];\n+\tstruct string_list symref = STRING_LIST_INIT_DUP;\n \n-\tif (!enter_repo(dir, strict))\n-\t\tdie(\"'%s' does not appear to be a git repository\", dir);\n+\tstateless_rpc = options->stateless_rpc;\n+\ttimeout = options->timeout;\n+\tdaemon_mode = options->daemon_mode;\n \n \tgit_config(upload_pack_config, NULL);\n \n-\tswitch (determine_protocol_version_server()) {\n-\tcase protocol_v1:\n-\t\t/*\n-\t\t * v1 is just the original protocol with a version string,\n-\t\t * so just fall through after writing the version string.\n-\t\t */\n-\t\tif (advertise_refs || !stateless_rpc)\n-\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n-\n-\t\t/* fallthrough */\n-\tcase protocol_v0:\n-\t\tupload_pack();\n-\t\tbreak;\n-\tcase protocol_unknown_version:\n-\t\tBUG(\"unknown protocol version\");\n+\thead_ref_namespaced(find_symref, &symref);\n+\n+\tif (options->advertise_refs || !stateless_rpc) {\n+\t\treset_timeout();\n+\t\thead_ref_namespaced(send_ref, &symref);\n+\t\tfor_each_namespaced_ref(send_ref, &symref);\n+\t\tadvertise_shallow_grafts(1);\n+\t\tpacket_flush(1);\n+\t} else {\n+\t\thead_ref_namespaced(check_ref, NULL);\n+\t\tfor_each_namespaced_ref(check_ref, NULL);\n \t}\n+\tstring_list_clear(&symref, 1);\n+\tif (options->advertise_refs)\n+\t\treturn;\n \n-\treturn 0;\n+\treceive_needs();\n+\tif (want_obj.nr) {\n+\t\tget_common_commits();\n+\t\tcreate_pack_file();\n+\t}\n }\ndiff --git a/upload-pack.h b/upload-pack.h\nnew file mode 100644\nindex 000000000..a71e4dc7e\n--- /dev/null\n+++ b/upload-pack.h\n@@ -0,0 +1,13 @@\n+#ifndef UPLOAD_PACK_H\n+#define UPLOAD_PACK_H\n+\n+struct upload_pack_options {\n+\tint stateless_rpc;\n+\tint advertise_refs;\n+\tunsigned int timeout;\n+\tint daemon_mode;\n+};\n+\n+void upload_pack(struct upload_pack_options *options);\n+\n+#endif /* UPLOAD_PACK_H */\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340645","messageId":"20180228232252.102167-4-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 03/35] pkt-line: add delim packet support","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:20Z","receivedAt":"2018-02-28T23:23:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"One of the design goals of protocol-v2 is to improve the semantics of\nflush packets. Currently in protocol-v1, flush packets are used both to\nindicate a break in a list of packet lines as well as an indication that\none side has finished speaking. This makes it particularly difficult\nto implement proxies as a proxy would need to completely understand git\nprotocol instead of simply looking for a flush packet.\n\nTo do this, introduce the special deliminator packet '0001'. A delim\npacket can then be used as a deliminator between lists of packet lines\nwhile flush packets can be reserved to indicate the end of a response.\n\nDocumentation for how this packet will be used in protocol v2 will\nincluded in a future patch.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 17 +++++++++++++++++\n pkt-line.h | 3 +++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 6307fa4a3..87a24bd17 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -91,6 +91,12 @@ void packet_flush(int fd)\n \twrite_or_die(fd, \"0000\", 4);\n }\n \n+void packet_delim(int fd)\n+{\n+\tpacket_trace(\"0001\", 4, 1);\n+\twrite_or_die(fd, \"0001\", 4);\n+}\n+\n int packet_flush_gently(int fd)\n {\n \tpacket_trace(\"0000\", 4, 1);\n@@ -105,6 +111,12 @@ void packet_buf_flush(struct strbuf *buf)\n \tstrbuf_add(buf, \"0000\", 4);\n }\n \n+void packet_buf_delim(struct strbuf *buf)\n+{\n+\tpacket_trace(\"0001\", 4, 1);\n+\tstrbuf_add(buf, \"0001\", 4);\n+}\n+\n static void set_packet_header(char *buf, const int size)\n {\n \tstatic char hexchar[] = \"0123456789abcdef\";\n@@ -298,6 +310,9 @@ enum packet_read_status packet_read_with_status(int fd, char **src_buffer,\n \t} else if (!len) {\n \t\tpacket_trace(\"0000\", 4, 0);\n \t\treturn PACKET_READ_FLUSH;\n+\t} else if (len == 1) {\n+\t\tpacket_trace(\"0001\", 4, 0);\n+\t\treturn PACKET_READ_DELIM;\n \t} else if (len < 4) {\n \t\tdie(\"protocol error: bad line length %d\", len);\n \t}\n@@ -331,6 +346,7 @@ int packet_read(int fd, char **src_buffer, size_t *src_len,\n \t\tbreak;\n \tcase PACKET_READ_NORMAL:\n \t\tbreak;\n+\tcase PACKET_READ_DELIM:\n \tcase PACKET_READ_FLUSH:\n \t\tpktlen = 0;\n \t\tbreak;\n@@ -443,6 +459,7 @@ enum packet_read_status packet_reader_read(struct packet_reader *reader)\n \tcase PACKET_READ_NORMAL:\n \t\treader->line = reader->buffer;\n \t\tbreak;\n+\tcase PACKET_READ_DELIM:\n \tcase PACKET_READ_FLUSH:\n \t\treader->pktlen = 0;\n \t\treader->line = NULL;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex f2edfae9a..3f836f01a 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -20,8 +20,10 @@\n * side can't, we stay with pure read/write interfaces.\n */\n void packet_flush(int fd);\n+void packet_delim(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n+void packet_buf_delim(struct strbuf *buf);\n void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n@@ -75,6 +77,7 @@ enum packet_read_status {\n \tPACKET_READ_EOF = -1,\n \tPACKET_READ_NORMAL,\n \tPACKET_READ_FLUSH,\n+\tPACKET_READ_DELIM,\n };\n enum packet_read_status packet_read_with_status(int fd, char **src_buffer,\n \t\t\t\t\t\tsize_t *src_len, char *buffer,\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340646","messageId":"20180228232252.102167-6-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 05/35] upload-pack: factor out processing lines","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:22Z","receivedAt":"2018-02-28T23:23:39Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Factor out the logic for processing shallow, deepen, deepen_since, and\ndeepen_not lines into their own functions to simplify the\n'receive_needs()' function in addition to making it easier to reuse some\nof this logic when implementing protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n upload-pack.c | 113 +++++++++++++++++++++++++++++++++-----------------\n 1 file changed, 74 insertions(+), 39 deletions(-)\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 2ad73a98b..1e8a9e1ca 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -724,6 +724,75 @@ static void deepen_by_rev_list(int ac, const char **av,\n \tpacket_flush(1);\n }\n \n+static int process_shallow(const char *line, struct object_array *shallows)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"shallow \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tstruct object *object;\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"invalid shallow line: %s\", line);\n+\t\tobject = parse_object(&oid);\n+\t\tif (!object)\n+\t\t\treturn 1;\n+\t\tif (object->type != OBJ_COMMIT)\n+\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n+\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n+\t\t\tobject->flags |= CLIENT_SHALLOW;\n+\t\t\tadd_object_array(object, NULL, shallows);\n+\t\t}\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_deepen(const char *line, int *depth)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\t*depth = (int)strtol(arg, &end, 0);\n+\t\tif (!end || *end || *depth <= 0)\n+\t\t\tdie(\"Invalid deepen: %s\", line);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_deepen_since(const char *line, timestamp_t *deepen_since, int *deepen_rev_list)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen-since \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\t*deepen_since = parse_timestamp(arg, &end, 0);\n+\t\tif (!end || *end || !deepen_since ||\n+\t\t /* revisions.c's max_age -1 is special */\n+\t\t *deepen_since == -1)\n+\t\t\tdie(\"Invalid deepen-since: %s\", line);\n+\t\t*deepen_rev_list = 1;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+static int process_deepen_not(const char *line, struct string_list *deepen_not, int *deepen_rev_list)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"deepen-not \", &arg)) {\n+\t\tchar *ref = NULL;\n+\t\tstruct object_id oid;\n+\t\tif (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n+\t\t\tdie(\"git upload-pack: ambiguous deepen-not: %s\", line);\n+\t\tstring_list_append(deepen_not, ref);\n+\t\tfree(ref);\n+\t\t*deepen_rev_list = 1;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static void receive_needs(void)\n {\n \tstruct object_array shallows = OBJECT_ARRAY_INIT;\n@@ -745,49 +814,15 @@ static void receive_needs(void)\n \t\tif (!line)\n \t\t\tbreak;\n \n-\t\tif (skip_prefix(line, \"shallow \", &arg)) {\n-\t\t\tstruct object_id oid;\n-\t\t\tstruct object *object;\n-\t\t\tif (get_oid_hex(arg, &oid))\n-\t\t\t\tdie(\"invalid shallow line: %s\", line);\n-\t\t\tobject = parse_object(&oid);\n-\t\t\tif (!object)\n-\t\t\t\tcontinue;\n-\t\t\tif (object->type != OBJ_COMMIT)\n-\t\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n-\t\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n-\t\t\t\tobject->flags |= CLIENT_SHALLOW;\n-\t\t\t\tadd_object_array(object, NULL, &shallows);\n-\t\t\t}\n+\t\tif (process_shallow(line, &shallows))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen \", &arg)) {\n-\t\t\tchar *end = NULL;\n-\t\t\tdepth = strtol(arg, &end, 0);\n-\t\t\tif (!end || *end || depth <= 0)\n-\t\t\t\tdie(\"Invalid deepen: %s\", line);\n+\t\tif (process_deepen(line, &depth))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen-since \", &arg)) {\n-\t\t\tchar *end = NULL;\n-\t\t\tdeepen_since = parse_timestamp(arg, &end, 0);\n-\t\t\tif (!end || *end || !deepen_since ||\n-\t\t\t /* revisions.c's max_age -1 is special */\n-\t\t\t deepen_since == -1)\n-\t\t\t\tdie(\"Invalid deepen-since: %s\", line);\n-\t\t\tdeepen_rev_list = 1;\n+\t\tif (process_deepen_since(line, &deepen_since, &deepen_rev_list))\n \t\t\tcontinue;\n-\t\t}\n-\t\tif (skip_prefix(line, \"deepen-not \", &arg)) {\n-\t\t\tchar *ref = NULL;\n-\t\t\tstruct object_id oid;\n-\t\t\tif (expand_ref(arg, strlen(arg), &oid, &ref) != 1)\n-\t\t\t\tdie(\"git upload-pack: ambiguous deepen-not: %s\", line);\n-\t\t\tstring_list_append(&deepen_not, ref);\n-\t\t\tfree(ref);\n-\t\t\tdeepen_rev_list = 1;\n+\t\tif (process_deepen_not(line, &deepen_not, &deepen_rev_list))\n \t\t\tcontinue;\n-\t\t}\n+\n \t\tif (!skip_prefix(line, \"want \", &arg) ||\n \t\t get_oid_hex(arg, &oid_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340647","messageId":"20180228232252.102167-7-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 06/35] transport: use get_refs_via_connect to get refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:23Z","receivedAt":"2018-02-28T23:23:41Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Remove code duplication and use the existing 'get_refs_via_connect()'\nfunction to retrieve a remote's heads in 'fetch_refs_via_pack()' and\n'git_transport_push()'.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 18 ++++--------------\n 1 file changed, 4 insertions(+), 14 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex fc802260f..8e8779096 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -230,12 +230,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n \n-\tif (!data->got_remote_heads) {\n-\t\tconnect_setup(transport, 0);\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL, &data->shallow);\n-\t\tdata->got_remote_heads = 1;\n-\t}\n+\tif (!data->got_remote_heads)\n+\t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n \trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n@@ -541,14 +537,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tstruct send_pack_args args;\n \tint ret;\n \n-\tif (!data->got_remote_heads) {\n-\t\tstruct ref *tmp_refs;\n-\t\tconnect_setup(transport, 1);\n-\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n-\t\t\t\t NULL, &data->shallow);\n-\t\tdata->got_remote_heads = 1;\n-\t}\n+\tif (!data->got_remote_heads)\n+\t\tget_refs_via_connect(transport, 1);\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340648","messageId":"20180228232252.102167-8-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 07/35] connect: convert get_remote_heads to use struct packet_reader","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:24Z","receivedAt":"2018-02-28T23:23:45Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to allow for better control flow when protocol_v2 is introduced\nconvert 'get_remote_heads()' to use 'struct packet_reader' to read\npacket lines. This enables a client to be able to peek the first line\nof a server's response (without consuming it) in order to determine the\nprotocol version its speaking and then passing control to the\nappropriate handler.\n\nThis is needed because the initial response from a server speaking\nprotocol_v0 includes the first ref, while subsequent protocol versions\nrespond with a version line. We want to be able to read this first line\nwithout consuming the first ref sent in the protocol_v0 case so that the\nprotocol version the server is speaking can be determined outside of\n'get_remote_heads()' in a future patch.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 173 ++++++++++++++++++++++++++++++------------------------\n 1 file changed, 95 insertions(+), 78 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex c3a014c5b..c82c90b7c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -48,6 +48,12 @@ int check_ref_type(const struct ref *ref, int flags)\n \n static void die_initial_contact(int unexpected)\n {\n+\t/*\n+\t * A hang-up after seeing some response from the other end\n+\t * means that it is unexpected, as we know the other end is\n+\t * willing to talk to us. A hang-up before seeing any\n+\t * response does not necessarily mean an ACL problem, though.\n+\t */\n \tif (unexpected)\n \t\tdie(_(\"The remote end hung up upon initial contact\"));\n \telse\n@@ -56,6 +62,40 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n+static enum protocol_version discover_version(struct packet_reader *reader)\n+{\n+\tenum protocol_version version = protocol_unknown_version;\n+\n+\t/*\n+\t * Peek the first line of the server's response to\n+\t * determine the protocol version the server is speaking.\n+\t */\n+\tswitch (packet_reader_peek(reader)) {\n+\tcase PACKET_READ_EOF:\n+\t\tdie_initial_contact(0);\n+\tcase PACKET_READ_FLUSH:\n+\tcase PACKET_READ_DELIM:\n+\t\tversion = protocol_v0;\n+\t\tbreak;\n+\tcase PACKET_READ_NORMAL:\n+\t\tversion = determine_protocol_version_client(reader->line);\n+\t\tbreak;\n+\t}\n+\n+\tswitch (version) {\n+\tcase protocol_v1:\n+\t\t/* Read the peeked version line */\n+\t\tpacket_reader_read(reader);\n+\t\tbreak;\n+\tcase protocol_v0:\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n+\treturn version;\n+}\n+\n static void parse_one_symref_info(struct string_list *symref, const char *val, int len)\n {\n \tchar *sym, *target;\n@@ -109,60 +149,21 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n-/*\n- * Read one line of a server's ref advertisement into packet_buffer.\n- */\n-static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n-\t\t\t int *responded)\n+static void process_capabilities(const char *line, int *len)\n {\n-\tint len = packet_read(in, src_buf, src_len,\n-\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\tconst char *arg;\n-\tif (len < 0)\n-\t\tdie_initial_contact(*responded);\n-\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n-\t\tdie(\"remote error: %s\", arg);\n-\n-\t*responded = 1;\n-\n-\treturn len;\n-}\n-\n-#define EXPECTING_PROTOCOL_VERSION 0\n-#define EXPECTING_FIRST_REF 1\n-#define EXPECTING_REF 2\n-#define EXPECTING_SHALLOW 3\n-\n-/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n-static int process_protocol_version(void)\n-{\n-\tswitch (determine_protocol_version_client(packet_buffer)) {\n-\tcase protocol_v1:\n-\t\treturn 1;\n-\tcase protocol_v0:\n-\t\treturn 0;\n-\tdefault:\n-\t\tdie(\"server is speaking an unknown protocol\");\n-\t}\n-}\n-\n-static void process_capabilities(int *len)\n-{\n-\tint nul_location = strlen(packet_buffer);\n+\tint nul_location = strlen(line);\n \tif (nul_location == *len)\n \t\treturn;\n-\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n+\tserver_capabilities = xstrdup(line + nul_location + 1);\n \t*len = nul_location;\n }\n \n-static int process_dummy_ref(void)\n+static int process_dummy_ref(const char *line)\n {\n \tstruct object_id oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(packet_buffer, &oid, &name))\n+\tif (parse_oid_hex(line, &oid, &name))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -171,20 +172,20 @@ static int process_dummy_ref(void)\n \treturn !oidcmp(&null_oid, &oid) && !strcmp(name, \"capabilities^{}\");\n }\n \n-static void check_no_capabilities(int len)\n+static void check_no_capabilities(const char *line, int len)\n {\n-\tif (strlen(packet_buffer) != len)\n+\tif (strlen(line) != len)\n \t\twarning(\"Ignoring capabilities after first line '%s'\",\n-\t\t\tpacket_buffer + strlen(packet_buffer));\n+\t\t\tline + strlen(line));\n }\n \n-static int process_ref(int len, struct ref ***list, unsigned int flags,\n-\t\t struct oid_array *extra_have)\n+static int process_ref(const char *line, int len, struct ref ***list,\n+\t\t unsigned int flags, struct oid_array *extra_have)\n {\n \tstruct object_id old_oid;\n \tconst char *name;\n \n-\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n+\tif (parse_oid_hex(line, &old_oid, &name))\n \t\treturn 0;\n \tif (*name != ' ')\n \t\treturn 0;\n@@ -200,16 +201,17 @@ static int process_ref(int len, struct ref ***list, unsigned int flags,\n \t\t**list = ref;\n \t\t*list = &ref->next;\n \t}\n-\tcheck_no_capabilities(len);\n+\tcheck_no_capabilities(line, len);\n \treturn 1;\n }\n \n-static int process_shallow(int len, struct oid_array *shallow_points)\n+static int process_shallow(const char *line, int len,\n+\t\t\t struct oid_array *shallow_points)\n {\n \tconst char *arg;\n \tstruct object_id old_oid;\n \n-\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\tif (!skip_prefix(line, \"shallow \", &arg))\n \t\treturn 0;\n \n \tif (get_oid_hex(arg, &old_oid))\n@@ -217,10 +219,17 @@ static int process_shallow(int len, struct oid_array *shallow_points)\n \tif (!shallow_points)\n \t\tdie(\"repository on the other end cannot be shallow\");\n \toid_array_append(shallow_points, &old_oid);\n-\tcheck_no_capabilities(len);\n+\tcheck_no_capabilities(line, len);\n \treturn 1;\n }\n \n+enum get_remote_heads_state {\n+\tEXPECTING_FIRST_REF = 0,\n+\tEXPECTING_REF,\n+\tEXPECTING_SHALLOW,\n+\tEXPECTING_DONE,\n+};\n+\n /*\n * Read all the refs from the other end\n */\n@@ -230,47 +239,55 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t struct oid_array *shallow_points)\n {\n \tstruct ref **orig_list = list;\n+\tint len = 0;\n+\tenum get_remote_heads_state state = EXPECTING_FIRST_REF;\n+\tstruct packet_reader reader;\n+\tconst char *arg;\n \n-\t/*\n-\t * A hang-up after seeing some response from the other end\n-\t * means that it is unexpected, as we know the other end is\n-\t * willing to talk to us. A hang-up before seeing any\n-\t * response does not necessarily mean an ACL problem, though.\n-\t */\n-\tint responded = 0;\n-\tint len;\n-\tint state = EXPECTING_PROTOCOL_VERSION;\n+\tpacket_reader_init(&reader, in, src_buf, src_len,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tdiscover_version(&reader);\n \n \t*list = NULL;\n \n-\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\twhile (state != EXPECTING_DONE) {\n+\t\tswitch (packet_reader_read(&reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tdie_initial_contact(1);\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tlen = reader.pktlen;\n+\t\t\tif (len > 4 && skip_prefix(reader.line, \"ERR \", &arg))\n+\t\t\t\tdie(\"remote error: %s\", arg);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tstate = EXPECTING_DONE;\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tdie(\"invalid packet\");\n+\t\t}\n+\n \t\tswitch (state) {\n-\t\tcase EXPECTING_PROTOCOL_VERSION:\n-\t\t\tif (process_protocol_version()) {\n-\t\t\t\tstate = EXPECTING_FIRST_REF;\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tstate = EXPECTING_FIRST_REF;\n-\t\t\t/* fallthrough */\n \t\tcase EXPECTING_FIRST_REF:\n-\t\t\tprocess_capabilities(&len);\n-\t\t\tif (process_dummy_ref()) {\n+\t\t\tprocess_capabilities(reader.line, &len);\n+\t\t\tif (process_dummy_ref(reader.line)) {\n \t\t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tstate = EXPECTING_REF;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_REF:\n-\t\t\tif (process_ref(len, &list, flags, extra_have))\n+\t\t\tif (process_ref(reader.line, len, &list, flags, extra_have))\n \t\t\t\tbreak;\n \t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_SHALLOW:\n-\t\t\tif (process_shallow(len, shallow_points))\n+\t\t\tif (process_shallow(reader.line, len, shallow_points))\n \t\t\t\tbreak;\n-\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n-\t\tdefault:\n-\t\t\tdie(\"unexpected state %d\", state);\n+\t\t\tdie(\"protocol error: unexpected '%s'\", reader.line);\n+\t\tcase EXPECTING_DONE:\n+\t\t\tbreak;\n \t\t}\n \t}\n \n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340649","messageId":"20180228232252.102167-9-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 08/35] connect: discover protocol version outside of get_remote_heads","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:25Z","receivedAt":"2018-02-28T23:23:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to prepare for the addition of protocol_v2 push the protocol\nversion discovery outside of 'get_remote_heads()'. This will allow for\nkeeping the logic for processing the reference advertisement for\nprotocol_v1 and protocol_v0 separate from the logic for protocol_v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch-pack.c | 16 +++++++++++++++-\n builtin/send-pack.c | 17 +++++++++++++++--\n connect.c | 27 ++++++++++-----------------\n connect.h | 3 +++\n remote-curl.c | 20 ++++++++++++++++++--\n remote.h | 5 +++--\n transport.c | 24 +++++++++++++++++++-----\n 7 files changed, 83 insertions(+), 29 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 366b9d13f..85d4faf76 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -4,6 +4,7 @@\n #include \"remote.h\"\n #include \"connect.h\"\n #include \"sha1-array.h\"\n+#include \"protocol.h\"\n \n static const char fetch_pack_usage[] =\n \"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] \"\n@@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tstruct fetch_pack_args args;\n \tstruct oid_array shallow = OID_ARRAY_INIT;\n \tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n+\tstruct packet_reader reader;\n \n \tpacket_trace_identity(\"fetch-pack\");\n \n@@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\tif (!conn)\n \t\t\treturn args.diag_url ? 0 : 1;\n \t}\n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n+\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n \t\t\t &shallow, pack_lockfile_ptr);\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex fc4f0bb5f..83cb125a6 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -14,6 +14,7 @@\n #include \"sha1-array.h\"\n #include \"gpg-interface.h\"\n #include \"gettext.h\"\n+#include \"protocol.h\"\n \n static const char * const send_pack_usage[] = {\n \tN_(\"git send-pack [--all | --mirror] [--dry-run] [--force] \"\n@@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint progress = -1;\n \tint from_stdin = 0;\n \tstruct push_cas_option cas = {0};\n+\tstruct packet_reader reader;\n \n \tstruct option options[] = {\n \t\tOPT__VERBOSITY(&verbose),\n@@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n-\t\t\t &extra_have, &shallow);\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n+\t\t\t\t &extra_have, &shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/connect.c b/connect.c\nindex c82c90b7c..0b111e62d 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n-static enum protocol_version discover_version(struct packet_reader *reader)\n+enum protocol_version discover_version(struct packet_reader *reader)\n {\n \tenum protocol_version version = protocol_unknown_version;\n \n@@ -233,7 +233,7 @@ enum get_remote_heads_state {\n /*\n * Read all the refs from the other end\n */\n-struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n+struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t struct ref **list, unsigned int flags,\n \t\t\t struct oid_array *extra_have,\n \t\t\t struct oid_array *shallow_points)\n@@ -241,24 +241,17 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \tstruct ref **orig_list = list;\n \tint len = 0;\n \tenum get_remote_heads_state state = EXPECTING_FIRST_REF;\n-\tstruct packet_reader reader;\n \tconst char *arg;\n \n-\tpacket_reader_init(&reader, in, src_buf, src_len,\n-\t\t\t PACKET_READ_CHOMP_NEWLINE |\n-\t\t\t PACKET_READ_GENTLE_ON_EOF);\n-\n-\tdiscover_version(&reader);\n-\n \t*list = NULL;\n \n \twhile (state != EXPECTING_DONE) {\n-\t\tswitch (packet_reader_read(&reader)) {\n+\t\tswitch (packet_reader_read(reader)) {\n \t\tcase PACKET_READ_EOF:\n \t\t\tdie_initial_contact(1);\n \t\tcase PACKET_READ_NORMAL:\n-\t\t\tlen = reader.pktlen;\n-\t\t\tif (len > 4 && skip_prefix(reader.line, \"ERR \", &arg))\n+\t\t\tlen = reader->pktlen;\n+\t\t\tif (len > 4 && skip_prefix(reader->line, \"ERR \", &arg))\n \t\t\t\tdie(\"remote error: %s\", arg);\n \t\t\tbreak;\n \t\tcase PACKET_READ_FLUSH:\n@@ -270,22 +263,22 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \n \t\tswitch (state) {\n \t\tcase EXPECTING_FIRST_REF:\n-\t\t\tprocess_capabilities(reader.line, &len);\n-\t\t\tif (process_dummy_ref(reader.line)) {\n+\t\t\tprocess_capabilities(reader->line, &len);\n+\t\t\tif (process_dummy_ref(reader->line)) {\n \t\t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tstate = EXPECTING_REF;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_REF:\n-\t\t\tif (process_ref(reader.line, len, &list, flags, extra_have))\n+\t\t\tif (process_ref(reader->line, len, &list, flags, extra_have))\n \t\t\t\tbreak;\n \t\t\tstate = EXPECTING_SHALLOW;\n \t\t\t/* fallthrough */\n \t\tcase EXPECTING_SHALLOW:\n-\t\t\tif (process_shallow(reader.line, len, shallow_points))\n+\t\t\tif (process_shallow(reader->line, len, shallow_points))\n \t\t\t\tbreak;\n-\t\t\tdie(\"protocol error: unexpected '%s'\", reader.line);\n+\t\t\tdie(\"protocol error: unexpected '%s'\", reader->line);\n \t\tcase EXPECTING_DONE:\n \t\t\tbreak;\n \t\t}\ndiff --git a/connect.h b/connect.h\nindex 01f14cdf3..cdb8979dc 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);\n extern const char *server_feature_value(const char *feature, int *len_ret);\n extern int url_is_local_not_ssh(const char *url);\n \n+struct packet_reader;\n+extern enum protocol_version discover_version(struct packet_reader *reader);\n+\n #endif\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 0053b0954..9f6d07683 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"remote.h\"\n+#include \"connect.h\"\n #include \"strbuf.h\"\n #include \"walker.h\"\n #include \"http.h\"\n@@ -13,6 +14,7 @@\n #include \"credential.h\"\n #include \"sha1-array.h\"\n #include \"send-pack.h\"\n+#include \"protocol.h\"\n \n static struct remote *remote;\n /* always ends with a trailing slash */\n@@ -176,8 +178,22 @@ static struct discovery *last_discovery;\n static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n-\tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n+\tstruct packet_reader reader;\n+\n+\tpacket_reader_init(&reader, -1, heads->buf, heads->len,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n+\t\t\t\t NULL, &heads->shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \treturn list;\n }\n \ndiff --git a/remote.h b/remote.h\nindex 1f6611be2..2016461df 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n void free_refs(struct ref *ref);\n \n struct oid_array;\n-extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n+struct packet_reader;\n+extern struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t\t struct ref **list, unsigned int flags,\n \t\t\t\t struct oid_array *extra_have,\n-\t\t\t\t struct oid_array *shallow);\n+\t\t\t\t struct oid_array *shallow_points);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\ndiff --git a/transport.c b/transport.c\nindex 8e8779096..63c3dbab9 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -18,6 +18,7 @@\n #include \"sha1-array.h\"\n #include \"sigchain.h\"\n #include \"transport-internal.h\"\n+#include \"protocol.h\"\n \n static void set_upstreams(struct transport *transport, struct ref *refs,\n \tint pretend)\n@@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)\n static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n {\n \tstruct git_transport_data *data = transport->data;\n-\tstruct ref *refs;\n+\tstruct ref *refs = NULL;\n+\tstruct packet_reader reader;\n \n \tconnect_setup(transport, for_push);\n-\tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0,\n-\t\t\t &data->extra_have,\n-\t\t\t &data->shallow);\n+\n+\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tswitch (discover_version(&reader)) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tget_remote_heads(&reader, &refs,\n+\t\t\t\t for_push ? REF_NORMAL : 0,\n+\t\t\t\t &data->extra_have,\n+\t\t\t\t &data->shallow);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340650","messageId":"20180228232252.102167-10-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 09/35] transport: store protocol version","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:26Z","receivedAt":"2018-02-28T23:23:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Once protocol_v2 is introduced requesting a fetch or a push will need to\nbe handled differently depending on the protocol version. Store the\nprotocol version the server is speaking in 'struct git_transport_data'\nand use it to determine what to do in the case of a fetch or a push.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 35 ++++++++++++++++++++++++++---------\n 1 file changed, 26 insertions(+), 9 deletions(-)\n\ndiff --git a/transport.c b/transport.c\nindex 63c3dbab9..2378dcb38 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -118,6 +118,7 @@ struct git_transport_data {\n \tstruct child_process *conn;\n \tint fd[2];\n \tunsigned got_remote_heads : 1;\n+\tenum protocol_version version;\n \tstruct oid_array extra_have;\n \tstruct oid_array shallow;\n };\n@@ -200,7 +201,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n-\tswitch (discover_version(&reader)) {\n+\tdata->version = discover_version(&reader);\n+\tswitch (data->version) {\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &refs,\n@@ -221,7 +223,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n {\n \tint ret = 0;\n \tstruct git_transport_data *data = transport->data;\n-\tstruct ref *refs;\n+\tstruct ref *refs = NULL;\n \tchar *dest = xstrdup(transport->url);\n \tstruct fetch_pack_args args;\n \tstruct ref *refs_tmp = NULL;\n@@ -247,10 +249,18 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tif (!data->got_remote_heads)\n \t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n-\trefs = fetch_pack(&args, data->fd, data->conn,\n-\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n-\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n-\t\t\t &transport->pack_lockfile);\n+\tswitch (data->version) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\trefs = fetch_pack(&args, data->fd, data->conn,\n+\t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n+\t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n+\t\t\t\t &transport->pack_lockfile);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \tclose(data->fd[0]);\n \tclose(data->fd[1]);\n \tif (finish_connect(data->conn))\n@@ -549,7 +559,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n {\n \tstruct git_transport_data *data = transport->data;\n \tstruct send_pack_args args;\n-\tint ret;\n+\tint ret = 0;\n \n \tif (!data->got_remote_heads)\n \t\tget_refs_via_connect(transport, 1);\n@@ -574,8 +584,15 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \telse\n \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n \n-\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n-\t\t\t&data->extra_have);\n+\tswitch (data->version) {\n+\tcase protocol_v1:\n+\tcase protocol_v0:\n+\t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n+\t\t\t\t&data->extra_have);\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n \n \tclose(data->fd[1]);\n \tclose(data->fd[0]);\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340651","messageId":"20180228232252.102167-12-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 11/35] test-pkt-line: introduce a packet-line test helper","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:28Z","receivedAt":"2018-02-28T23:23:53Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce a packet-line test helper which can either pack or unpack an\ninput stream into packet-lines and writes out the result to stdout.\n\nSigned-off-by: Brandon Williams \n---\n Makefile | 1 +\n t/helper/test-pkt-line.c | 64 ++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 65 insertions(+)\n create mode 100644 t/helper/test-pkt-line.c\n\ndiff --git a/Makefile b/Makefile\nindex b7ccc05fa..3b849c060 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -669,6 +669,7 @@ TEST_PROGRAMS_NEED_X += test-mktemp\n TEST_PROGRAMS_NEED_X += test-online-cpus\n TEST_PROGRAMS_NEED_X += test-parse-options\n TEST_PROGRAMS_NEED_X += test-path-utils\n+TEST_PROGRAMS_NEED_X += test-pkt-line\n TEST_PROGRAMS_NEED_X += test-prio-queue\n TEST_PROGRAMS_NEED_X += test-read-cache\n TEST_PROGRAMS_NEED_X += test-write-cache\ndiff --git a/t/helper/test-pkt-line.c b/t/helper/test-pkt-line.c\nnew file mode 100644\nindex 000000000..0f19e53c7\n--- /dev/null\n+++ b/t/helper/test-pkt-line.c\n@@ -0,0 +1,64 @@\n+#include \"pkt-line.h\"\n+\n+static void pack_line(const char *line)\n+{\n+\tif (!strcmp(line, \"0000\") || !strcmp(line, \"0000\\n\"))\n+\t\tpacket_flush(1);\n+\telse if (!strcmp(line, \"0001\") || !strcmp(line, \"0001\\n\"))\n+\t\tpacket_delim(1);\n+\telse\n+\t\tpacket_write_fmt(1, \"%s\", line);\n+}\n+\n+static void pack(int argc, const char **argv)\n+{\n+\tif (argc) { /* read from argv */\n+\t\tint i;\n+\t\tfor (i = 0; i < argc; i++)\n+\t\t\tpack_line(argv[i]);\n+\t} else { /* read from stdin */\n+\t\tchar line[LARGE_PACKET_MAX];\n+\t\twhile (fgets(line, sizeof(line), stdin)) {\n+\t\t\tpack_line(line);\n+\t\t}\n+\t}\n+}\n+\n+static void unpack(void)\n+{\n+\tstruct packet_reader reader;\n+\tpacket_reader_init(&reader, 0, NULL, 0,\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (packet_reader_read(&reader) != PACKET_READ_EOF) {\n+\t\tswitch (reader.status) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tprintf(\"%s\\n\", reader.line);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tprintf(\"0000\\n\");\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tprintf(\"0001\\n\");\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tif (argc < 2)\n+\t\tdie(\"too few arguments\");\n+\n+\tif (!strcmp(argv[1], \"pack\"))\n+\t\tpack(argc - 2, argv + 2);\n+\telse if (!strcmp(argv[1], \"unpack\"))\n+\t\tunpack();\n+\telse\n+\t\tdie(\"invalid argument '%s'\", argv[1]);\n+\n+\treturn 0;\n+}\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340652","messageId":"20180228232252.102167-13-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 12/35] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:29Z","receivedAt":"2018-02-28T23:23:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce git-serve, the base server for protocol version 2.\n\nProtocol version 2 is intended to be a replacement for Git's current\nwire protocol. The intention is that it will be a simpler, less\nwasteful protocol which can evolve over time.\n\nProtocol version 2 improves upon version 1 by eliminating the initial\nref advertisement. In its place a server will export a list of\ncapabilities and commands which it supports in a capability\nadvertisement. A client can then request that a particular command be\nexecuted by providing a number of capabilities and command specific\nparameters. At the completion of a command, a client can request that\nanother command be executed or can terminate the connection by sending a\nflush packet.\n\nSigned-off-by: Brandon Williams \n---\n .gitignore | 1 +\n Documentation/technical/protocol-v2.txt | 171 ++++++++++++++++\n Makefile | 2 +\n builtin.h | 1 +\n builtin/serve.c | 30 +++\n git.c | 1 +\n serve.c | 250 ++++++++++++++++++++++++\n serve.h | 15 ++\n t/t5701-git-serve.sh | 60 ++++++\n 9 files changed, 531 insertions(+)\n create mode 100644 Documentation/technical/protocol-v2.txt\n create mode 100644 builtin/serve.c\n create mode 100644 serve.c\n create mode 100644 serve.h\n create mode 100755 t/t5701-git-serve.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 833ef3b0b..2d0450c26 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -140,6 +140,7 @@\n /git-rm\n /git-send-email\n /git-send-pack\n+/git-serve\n /git-sh-i18n\n /git-sh-i18n--envsubst\n /git-sh-setup\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nnew file mode 100644\nindex 000000000..b02eefc21\n--- /dev/null\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -0,0 +1,171 @@\n+ Git Wire Protocol, Version 2\n+==============================\n+\n+This document presents a specification for a version 2 of Git's wire\n+protocol. Protocol v2 will improve upon v1 in the following ways:\n+\n+ * Instead of multiple service names, multiple commands will be\n+ supported by a single service\n+ * Easily extendable as capabilities are moved into their own section\n+ of the protocol, no longer being hidden behind a NUL byte and\n+ limited by the size of a pkt-line\n+ * Separate out other information hidden behind NUL bytes (e.g. agent\n+ string as a capability and symrefs can be requested using 'ls-refs')\n+ * Reference advertisement will be omitted unless explicitly requested\n+ * ls-refs command to explicitly request some refs\n+ * Designed with http and stateless-rpc in mind. With clear flush\n+ semantics the http remote helper can simply act as a proxy\n+\n+ Detailed Design\n+=================\n+\n+In protocol v2 communication is command oriented. When first contacting a\n+server a list of capabilities will advertised. Some of these capabilities\n+will be commands which a client can request be executed. Once a command\n+has completed, a client can reuse the connection and request that other\n+commands be executed.\n+\n+ Packet-Line Framing\n+---------------------\n+\n+All communication is done using packet-line framing, just as in v1. See\n+`Documentation/technical/pack-protocol.txt` and\n+`Documentation/technical/protocol-common.txt` for more information.\n+\n+In protocol v2 these special packets will have the following semantics:\n+\n+ * '0000' Flush Packet (flush-pkt) - indicates the end of a message\n+ * '0001' Delimiter Packet (delim-pkt) - separates sections of a message\n+\n+ Initial Client Request\n+------------------------\n+\n+In general a client can request to speak protocol v2 by sending\n+`version=2` through the respective side-channel for the transport being\n+used which inevitably sets `GIT_PROTOCOL`. More information can be\n+found in `pack-protocol.txt` and `http-protocol.txt`. In all cases the\n+response from the server is the capability advertisement.\n+\n+ Git Transport\n+~~~~~~~~~~~~~~~\n+\n+When using the git:// transport, you can request to use protocol v2 by\n+sending \"version=2\" as an extra parameter:\n+\n+ 003egit-upload-pack /project.git\\0host=myserver.com\\0\\0version=2\\0\n+\n+ SSH and File Transport\n+~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+When using either the ssh:// or file:// transport, the GIT_PROTOCOL\n+environment variable must be set explicitly to include \"version=2\".\n+\n+ HTTP Transport\n+~~~~~~~~~~~~~~~~\n+\n+When using the http:// or https:// transport a client makes a \"smart\"\n+info/refs request as described in `http-protocol.txt` and requests that\n+v2 be used by supplying \"version=2\" in the `Git-Protocol` header.\n+\n+ C: Git-Protocol: version=2\n+ C:\n+ C: GET $GIT_URL/info/refs?service=git-upload-pack HTTP/1.0\n+\n+A v2 server would reply:\n+\n+ S: 200 OK\n+ S: \n+ S: ...\n+ S:\n+ S: 000eversion 2\\n\n+ S: \n+\n+Subsequent requests are then made directly to the service\n+`$GIT_URL/git-upload-pack`. (This works the same for git-receive-pack).\n+\n+ Capability Advertisement\n+--------------------------\n+\n+A server which decides to communicate (based on a request from a client)\n+using protocol version 2, notifies the client by sending a version string\n+in its initial response followed by an advertisement of its capabilities.\n+Each capability is a key with an optional value. Clients must ignore all\n+unknown keys. Semantics of unknown values are left to the definition of\n+each key. Some capabilities will describe commands which can be requested\n+to be executed by the client.\n+\n+ capability-advertisement = protocol-version\n+\t\t\t capability-list\n+\t\t\t flush-pkt\n+\n+ protocol-version = PKT-LINE(\"version 2\" LF)\n+ capability-list = *capability\n+ capability = PKT-LINE(key[=value] LF)\n+\n+ key = 1*(ALPHA | DIGIT | \"-_\")\n+ value = 1*(ALPHA | DIGIT | \" -_.,?\\/{}[]()<>!@#$%^&*+=:;\")\n+\n+ Command Request\n+-----------------\n+\n+After receiving the capability advertisement, a client can then issue a\n+request to select the command it wants with any particular capabilities\n+or arguments. There is then an optional section where the client can\n+provide any command specific parameters or queries. Only a single\n+command can be requested at a time.\n+\n+ request = empty-request | command-request\n+ empty-request = flush-pkt\n+ command-request = command\n+\t\t capability-list\n+\t\t (command-args)\n+\t\t flush-pkt\n+ command = PKT-LINE(\"command=\" key LF)\n+ command-args = delim-pkt\n+\t\t *PKT-Line(arg LF)\n+ arg = 1*(ALPHA | DIGIT | \" -_.,?\\/{}[]()<>!@#$%^&*+=:;\")\n+\n+The server will then check to ensure that the client's request is\n+comprised of a valid command as well as valid capabilities which were\n+advertised. If the request is valid the server will then execute the\n+command. A server MUST wait till it has received the client's entire\n+request before issuing a response. The format of the response is\n+determined by the command being executed, but in all cases a flush-pkt\n+indicates the end of the response.\n+\n+When a command has finished, and the client has received the entire\n+response from the server, a client can either request that another\n+command be executed or can terminate the connection. A client may\n+optionally send an empty request consisting of just a flush-pkt to\n+indicate that no more requests will be made.\n+\n+ Capabilities\n+~~~~~~~~~~~~~~\n+\n+There are two different types of capabilities: normal capabilities,\n+which can be used to to convey information or alter the behavior of a\n+request, and commands, which are the core actions that a client wants to\n+perform (fetch, push, etc).\n+\n+All commands must only last a single round and be stateless from the\n+perspective of the server side. All state MUST be retained and managed\n+by the client process. This permits simple round-robin load-balancing\n+on the server side, without needing to worry about state management.\n+\n+Clients MUST NOT require state management on the server side in order to\n+function correctly.\n+\n+ agent\n+-------\n+\n+The server can advertise the `agent` capability with a value `X` (in the\n+form `agent=X`) to notify the client that the server is running version\n+`X`. The client may optionally send its own agent string by including\n+the `agent` capability with a value `Y` (in the form `agent=Y`) in its\n+request to the server (but it MUST NOT do so if the server did not\n+advertise the agent capability). The `X` and `Y` strings may contain any\n+printable ASCII characters except space (i.e., the byte range 32 < x <\n+127), and are typically of the form \"package/version\" (e.g.,\n+\"git/1.8.3.1\"). The agent strings are purely informative for statistics\n+and debugging purposes, and MUST NOT be used to programmatically assume\n+the presence or absence of particular features.\ndiff --git a/Makefile b/Makefile\nindex 3b849c060..18c255428 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -881,6 +881,7 @@ LIB_OBJS += revision.o\n LIB_OBJS += run-command.o\n LIB_OBJS += send-pack.o\n LIB_OBJS += sequencer.o\n+LIB_OBJS += serve.o\n LIB_OBJS += server-info.o\n LIB_OBJS += setup.o\n LIB_OBJS += sha1-array.o\n@@ -1014,6 +1015,7 @@ BUILTIN_OBJS += builtin/rev-parse.o\n BUILTIN_OBJS += builtin/revert.o\n BUILTIN_OBJS += builtin/rm.o\n BUILTIN_OBJS += builtin/send-pack.o\n+BUILTIN_OBJS += builtin/serve.o\n BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-ref.o\ndiff --git a/builtin.h b/builtin.h\nindex f332a1257..3f3fdfc28 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -215,6 +215,7 @@ extern int cmd_rev_parse(int argc, const char **argv, const char *prefix);\n extern int cmd_revert(int argc, const char **argv, const char *prefix);\n extern int cmd_rm(int argc, const char **argv, const char *prefix);\n extern int cmd_send_pack(int argc, const char **argv, const char *prefix);\n+extern int cmd_serve(int argc, const char **argv, const char *prefix);\n extern int cmd_shortlog(int argc, const char **argv, const char *prefix);\n extern int cmd_show(int argc, const char **argv, const char *prefix);\n extern int cmd_show_branch(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/serve.c b/builtin/serve.c\nnew file mode 100644\nindex 000000000..d3fd240bb\n--- /dev/null\n+++ b/builtin/serve.c\n@@ -0,0 +1,30 @@\n+#include \"cache.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"serve.h\"\n+\n+static char const * const serve_usage[] = {\n+\tN_(\"git serve []\"),\n+\tNULL\n+};\n+\n+int cmd_serve(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct serve_options opts = SERVE_OPTIONS_INIT;\n+\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n+\t\t\t N_(\"quit after a single request/response exchange\")),\n+\t\tOPT_BOOL(0, \"advertise-capabilities\", &opts.advertise_capabilities,\n+\t\t\t N_(\"exit immediately after advertising capabilities\")),\n+\t\tOPT_END()\n+\t};\n+\n+\t/* ignore all unknown cmdline switches for now */\n+\targc = parse_options(argc, argv, prefix, options, serve_usage,\n+\t\t\t PARSE_OPT_KEEP_DASHDASH |\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\tserve(&opts);\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex f71073dc8..f85d682b6 100644\n--- a/git.c\n+++ b/git.c\n@@ -461,6 +461,7 @@ static struct cmd_struct commands[] = {\n \t{ \"revert\", cmd_revert, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"rm\", cmd_rm, RUN_SETUP },\n \t{ \"send-pack\", cmd_send_pack, RUN_SETUP },\n+\t{ \"serve\", cmd_serve, RUN_SETUP },\n \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n \t{ \"show\", cmd_show, RUN_SETUP },\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\ndiff --git a/serve.c b/serve.c\nnew file mode 100644\nindex 000000000..cf23179b9\n--- /dev/null\n+++ b/serve.c\n@@ -0,0 +1,250 @@\n+#include \"cache.h\"\n+#include \"repository.h\"\n+#include \"config.h\"\n+#include \"pkt-line.h\"\n+#include \"version.h\"\n+#include \"argv-array.h\"\n+#include \"serve.h\"\n+\n+static int agent_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, git_user_agent_sanitized());\n+\treturn 1;\n+}\n+\n+struct protocol_capability {\n+\t/*\n+\t * The name of the capability. The server uses this name when\n+\t * advertising this capability, and the client uses this name to\n+\t * specify this capability.\n+\t */\n+\tconst char *name;\n+\n+\t/*\n+\t * Function queried to see if a capability should be advertised.\n+\t * Optionally a value can be specified by adding it to 'value'.\n+\t * If a value is added to 'value', the server will advertise this\n+\t * capability as \"=\" instead of \"\".\n+\t */\n+\tint (*advertise)(struct repository *r, struct strbuf *value);\n+\n+\t/*\n+\t * Function called when a client requests the capability as a command.\n+\t * The command request will be provided to the function via 'keys', the\n+\t * capabilities requested, and 'args', the command specific parameters.\n+\t *\n+\t * This field should be NULL for capabilities which are not commands.\n+\t */\n+\tint (*command)(struct repository *r,\n+\t\t struct argv_array *keys,\n+\t\t struct argv_array *args);\n+};\n+\n+static struct protocol_capability capabilities[] = {\n+\t{ \"agent\", agent_advertise, NULL },\n+};\n+\n+static void advertise_capabilities(void)\n+{\n+\tstruct strbuf capability = STRBUF_INIT;\n+\tstruct strbuf value = STRBUF_INIT;\n+\tint i;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n+\t\tstruct protocol_capability *c = &capabilities[i];\n+\n+\t\tif (c->advertise(the_repository, &value)) {\n+\t\t\tstrbuf_addstr(&capability, c->name);\n+\n+\t\t\tif (value.len) {\n+\t\t\t\tstrbuf_addch(&capability, '=');\n+\t\t\t\tstrbuf_addbuf(&capability, &value);\n+\t\t\t}\n+\n+\t\t\tstrbuf_addch(&capability, '\\n');\n+\t\t\tpacket_write(1, capability.buf, capability.len);\n+\t\t}\n+\n+\t\tstrbuf_reset(&capability);\n+\t\tstrbuf_reset(&value);\n+\t}\n+\n+\tpacket_flush(1);\n+\tstrbuf_release(&capability);\n+\tstrbuf_release(&value);\n+}\n+\n+static struct protocol_capability *get_capability(const char *key)\n+{\n+\tint i;\n+\n+\tif (!key)\n+\t\treturn NULL;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n+\t\tstruct protocol_capability *c = &capabilities[i];\n+\t\tconst char *out;\n+\t\tif (skip_prefix(key, c->name, &out) && (!*out || *out == '='))\n+\t\t\treturn c;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static int is_valid_capability(const char *key)\n+{\n+\tconst struct protocol_capability *c = get_capability(key);\n+\n+\treturn c && c->advertise(the_repository, NULL);\n+}\n+\n+static int is_command(const char *key, struct protocol_capability **command)\n+{\n+\tconst char *out;\n+\n+\tif (skip_prefix(key, \"command=\", &out)) {\n+\t\tstruct protocol_capability *cmd = get_capability(out);\n+\n+\t\tif (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)\n+\t\t\tdie(\"invalid command '%s'\", out);\n+\t\tif (*command)\n+\t\t\tdie(\"command already requested\");\n+\n+\t\t*command = cmd;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int has_capability(const struct argv_array *keys, const char *capability,\n+\t\t const char **value)\n+{\n+\tint i;\n+\tfor (i = 0; i < keys->argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(keys->argv[i], capability, &out) &&\n+\t\t (!*out || *out == '=')) {\n+\t\t\tif (value) {\n+\t\t\t\tif (*out == '=')\n+\t\t\t\t\tout++;\n+\t\t\t\t*value = out;\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+enum request_state {\n+\tPROCESS_REQUEST_KEYS = 0,\n+\tPROCESS_REQUEST_ARGS,\n+\tPROCESS_REQUEST_DONE,\n+};\n+\n+static int process_request(void)\n+{\n+\tenum request_state state = PROCESS_REQUEST_KEYS;\n+\tstruct packet_reader reader;\n+\tstruct argv_array keys = ARGV_ARRAY_INIT;\n+\tstruct argv_array args = ARGV_ARRAY_INIT;\n+\tstruct protocol_capability *command = NULL;\n+\n+\tpacket_reader_init(&reader, 0, NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE |\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\t/*\n+\t * Check to see if the client closed their end before sending another\n+\t * request. If so we can terminate the connection.\n+\t */\n+\tif (packet_reader_peek(&reader) == PACKET_READ_EOF)\n+\t\treturn 1;\n+\treader.options = PACKET_READ_CHOMP_NEWLINE;\n+\n+\twhile (state != PROCESS_REQUEST_DONE) {\n+\t\tswitch (packet_reader_read(&reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tBUG(\"Should have already died when seeing EOF\");\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tstate = PROCESS_REQUEST_DONE;\n+\t\t\tcontinue;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tif (state != PROCESS_REQUEST_KEYS)\n+\t\t\t\tdie(\"protocol error\");\n+\t\t\tstate = PROCESS_REQUEST_ARGS;\n+\t\t\t/*\n+\t\t\t * maybe include a check to make sure that a\n+\t\t\t * command/capabilities were given.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tswitch (state) {\n+\t\tcase PROCESS_REQUEST_KEYS:\n+\t\t\t/* collect request; a sequence of keys and values */\n+\t\t\tif (is_command(reader.line, &command) ||\n+\t\t\t is_valid_capability(reader.line))\n+\t\t\t\targv_array_push(&keys, reader.line);\n+\t\t\telse\n+\t\t\t\tdie(\"unknown capability '%s'\", reader.line);\n+\t\t\tbreak;\n+\t\tcase PROCESS_REQUEST_ARGS:\n+\t\t\t/* collect arguments for the requested command */\n+\t\t\targv_array_push(&args, reader.line);\n+\t\t\tbreak;\n+\t\tcase PROCESS_REQUEST_DONE:\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If no command and no keys were given then the client wanted to\n+\t * terminate the connection.\n+\t */\n+\tif (!keys.argc && !args.argc)\n+\t\treturn 1;\n+\n+\tif (!command)\n+\t\tdie(\"no command requested\");\n+\n+\tcommand->command(the_repository, &keys, &args);\n+\n+\targv_array_clear(&keys);\n+\targv_array_clear(&args);\n+\treturn 0;\n+}\n+\n+/* Main serve loop for protocol version 2 */\n+void serve(struct serve_options *options)\n+{\n+\tif (options->advertise_capabilities || !options->stateless_rpc) {\n+\t\t/* serve by default supports v2 */\n+\t\tpacket_write_fmt(1, \"version 2\\n\");\n+\n+\t\tadvertise_capabilities();\n+\t\t/*\n+\t\t * If only the list of capabilities was requested exit\n+\t\t * immediately after advertising capabilities\n+\t\t */\n+\t\tif (options->advertise_capabilities)\n+\t\t\treturn;\n+\t}\n+\n+\t/*\n+\t * If stateless-rpc was requested then exit after\n+\t * a single request/response exchange\n+\t */\n+\tif (options->stateless_rpc) {\n+\t\tprocess_request();\n+\t} else {\n+\t\tfor (;;)\n+\t\t\tif (process_request())\n+\t\t\t\tbreak;\n+\t}\n+}\ndiff --git a/serve.h b/serve.h\nnew file mode 100644\nindex 000000000..fe65ba9f4\n--- /dev/null\n+++ b/serve.h\n@@ -0,0 +1,15 @@\n+#ifndef SERVE_H\n+#define SERVE_H\n+\n+struct argv_array;\n+extern int has_capability(const struct argv_array *keys, const char *capability,\n+\t\t\t const char **value);\n+\n+struct serve_options {\n+\tunsigned advertise_capabilities;\n+\tunsigned stateless_rpc;\n+};\n+#define SERVE_OPTIONS_INIT { 0 }\n+extern void serve(struct serve_options *options);\n+\n+#endif /* SERVE_H */\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nnew file mode 100755\nindex 000000000..affbad097\n--- /dev/null\n+++ b/t/t5701-git-serve.sh\n@@ -0,0 +1,60 @@\n+#!/bin/sh\n+\n+test_description='test git-serve and server commands'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'test capability advertisement' '\n+\tcat >expect <<-EOF &&\n+\tversion 2\n+\tagent=git/$(git version | cut -d\" \" -f3)\n+\t0000\n+\tEOF\n+\n+\tgit serve --advertise-capabilities >out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'stateless-rpc flag does not list capabilities' '\n+\t# Empty request\n+\ttest-pkt-line pack >in <<-EOF &&\n+\t0000\n+\tEOF\n+\tgit serve --stateless-rpc >out out &&\n+\ttest_must_be_empty out\n+'\n+\n+test_expect_success 'request invalid capability' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tfoobar\n+\t0000\n+\tEOF\n+\ttest_must_fail git serve --stateless-rpc 2>err in <<-EOF &&\n+\tagent=git/test\n+\t0000\n+\tEOF\n+\ttest_must_fail git serve --stateless-rpc 2>err in <<-EOF &&\n+\tcommand=foo\n+\tagent=git/test\n+\t0000\n+\tEOF\n+\ttest_must_fail git serve --stateless-rpc 2>err \n---\n builtin/fetch-pack.c | 2 ++\n builtin/receive-pack.c | 6 ++++++\n builtin/send-pack.c | 3 +++\n builtin/upload-pack.c | 7 +++++++\n connect.c | 3 +++\n protocol.c | 2 ++\n protocol.h | 1 +\n remote-curl.c | 3 +++\n transport.c | 9 +++++++++\n 9 files changed, 36 insertions(+)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 85d4faf76..b2374ddbb 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -201,6 +201,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex b7ce7c7f5..3656e94fd 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1963,6 +1963,12 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tunpack_limit = receive_unpack_limit;\n \n \tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v2:\n+\t\t/*\n+\t\t * push support for protocol v2 has not been implemented yet,\n+\t\t * so ignore the request to use v2 and fallback to using v0.\n+\t\t */\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 83cb125a6..b5427f75e 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -263,6 +263,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\ndiff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\nindex 2cb5cb35b..8d53e9794 100644\n--- a/builtin/upload-pack.c\n+++ b/builtin/upload-pack.c\n@@ -47,6 +47,13 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n \tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v2:\n+\t\t/*\n+\t\t * fetch support for protocol v2 has not been implemented yet,\n+\t\t * so ignore the request to use v2 and fallback to using v0.\n+\t\t */\n+\t\tupload_pack(&opts);\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\ndiff --git a/connect.c b/connect.c\nindex 0b111e62d..4b89b984c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -83,6 +83,9 @@ enum protocol_version discover_version(struct packet_reader *reader)\n \t}\n \n \tswitch (version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \t\t/* Read the peeked version line */\n \t\tpacket_reader_read(reader);\ndiff --git a/protocol.c b/protocol.c\nindex 43012b7eb..5e636785d 100644\n--- a/protocol.c\n+++ b/protocol.c\n@@ -8,6 +8,8 @@ static enum protocol_version parse_protocol_version(const char *value)\n \t\treturn protocol_v0;\n \telse if (!strcmp(value, \"1\"))\n \t\treturn protocol_v1;\n+\telse if (!strcmp(value, \"2\"))\n+\t\treturn protocol_v2;\n \telse\n \t\treturn protocol_unknown_version;\n }\ndiff --git a/protocol.h b/protocol.h\nindex 1b2bc94a8..2ad35e433 100644\n--- a/protocol.h\n+++ b/protocol.h\n@@ -5,6 +5,7 @@ enum protocol_version {\n \tprotocol_unknown_version = -1,\n \tprotocol_v0 = 0,\n \tprotocol_v1 = 1,\n+\tprotocol_v2 = 2,\n };\n \n /*\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 9f6d07683..dae8a4a48 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -185,6 +185,9 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n \tswitch (discover_version(&reader)) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\ndiff --git a/transport.c b/transport.c\nindex 2378dcb38..83d9dd1df 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -203,6 +203,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tget_remote_heads(&reader, &refs,\n@@ -250,6 +253,9 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\trefs_tmp = get_refs_via_connect(transport, 0);\n \n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\trefs = fetch_pack(&args, data->fd, data->conn,\n@@ -585,6 +591,9 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\targs.push_cert = SEND_PACK_PUSH_CERT_NEVER;\n \n \tswitch (data->version) {\n+\tcase protocol_v2:\n+\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\tret = send_pack(&args, data->fd, data->conn, remote_refs,\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340654","messageId":"20180228232252.102167-15-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 14/35] connect: request remote refs using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:31Z","receivedAt":"2018-02-28T23:24:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach the client to be able to request a remote's refs using protocol\nv2. This is done by having a client issue a 'ls-refs' request to a v2\nserver.\n\nSigned-off-by: Brandon Williams \n---\n builtin/upload-pack.c | 10 +--\n connect.c | 138 +++++++++++++++++++++++++++++++++++++++--\n connect.h | 2 +\n remote.h | 6 ++\n t/t5702-protocol-v2.sh | 57 +++++++++++++++++\n transport.c | 2 +-\n 6 files changed, 204 insertions(+), 11 deletions(-)\n create mode 100755 t/t5702-protocol-v2.sh\n\ndiff --git a/builtin/upload-pack.c b/builtin/upload-pack.c\nindex 8d53e9794..a757df8da 100644\n--- a/builtin/upload-pack.c\n+++ b/builtin/upload-pack.c\n@@ -5,6 +5,7 @@\n #include \"parse-options.h\"\n #include \"protocol.h\"\n #include \"upload-pack.h\"\n+#include \"serve.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -16,6 +17,7 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \tconst char *dir;\n \tint strict = 0;\n \tstruct upload_pack_options opts = { 0 };\n+\tstruct serve_options serve_opts = SERVE_OPTIONS_INIT;\n \tstruct option options[] = {\n \t\tOPT_BOOL(0, \"stateless-rpc\", &opts.stateless_rpc,\n \t\t\t N_(\"quit after a single request/response exchange\")),\n@@ -48,11 +50,9 @@ int cmd_upload_pack(int argc, const char **argv, const char *prefix)\n \n \tswitch (determine_protocol_version_server()) {\n \tcase protocol_v2:\n-\t\t/*\n-\t\t * fetch support for protocol v2 has not been implemented yet,\n-\t\t * so ignore the request to use v2 and fallback to using v0.\n-\t\t */\n-\t\tupload_pack(&opts);\n+\t\tserve_opts.advertise_capabilities = opts.advertise_refs;\n+\t\tserve_opts.stateless_rpc = opts.stateless_rpc;\n+\t\tserve(&serve_opts);\n \t\tbreak;\n \tcase protocol_v1:\n \t\t/*\ndiff --git a/connect.c b/connect.c\nindex 4b89b984c..6203ce576 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -12,9 +12,11 @@\n #include \"sha1-array.h\"\n #include \"transport.h\"\n #include \"strbuf.h\"\n+#include \"version.h\"\n #include \"protocol.h\"\n \n-static char *server_capabilities;\n+static char *server_capabilities_v1;\n+static struct argv_array server_capabilities_v2 = ARGV_ARRAY_INIT;\n static const char *parse_feature_value(const char *, const char *, int *);\n \n static int check_ref(const char *name, unsigned int flags)\n@@ -62,6 +64,33 @@ static void die_initial_contact(int unexpected)\n \t\t \"and the repository exists.\"));\n }\n \n+/* Checks if the server supports the capability 'c' */\n+int server_supports_v2(const char *c, int die_on_error)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n+\t\t (!*out || *out == '='))\n+\t\t\treturn 1;\n+\t}\n+\n+\tif (die_on_error)\n+\t\tdie(\"server doesn't support '%s'\", c);\n+\n+\treturn 0;\n+}\n+\n+static void process_capabilities_v2(struct packet_reader *reader)\n+{\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL)\n+\t\targv_array_push(&server_capabilities_v2, reader->line);\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\tdie(\"expected flush after capabilities\");\n+}\n+\n enum protocol_version discover_version(struct packet_reader *reader)\n {\n \tenum protocol_version version = protocol_unknown_version;\n@@ -84,7 +113,7 @@ enum protocol_version discover_version(struct packet_reader *reader)\n \n \tswitch (version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tprocess_capabilities_v2(reader);\n \t\tbreak;\n \tcase protocol_v1:\n \t\t/* Read the peeked version line */\n@@ -128,7 +157,7 @@ static void parse_one_symref_info(struct string_list *symref, const char *val, i\n static void annotate_refs_with_symref_info(struct ref *ref)\n {\n \tstruct string_list symref = STRING_LIST_INIT_DUP;\n-\tconst char *feature_list = server_capabilities;\n+\tconst char *feature_list = server_capabilities_v1;\n \n \twhile (feature_list) {\n \t\tint len;\n@@ -157,7 +186,7 @@ static void process_capabilities(const char *line, int *len)\n \tint nul_location = strlen(line);\n \tif (nul_location == *len)\n \t\treturn;\n-\tserver_capabilities = xstrdup(line + nul_location + 1);\n+\tserver_capabilities_v1 = xstrdup(line + nul_location + 1);\n \t*len = nul_location;\n }\n \n@@ -292,6 +321,105 @@ struct ref **get_remote_heads(struct packet_reader *reader,\n \treturn list;\n }\n \n+/* Returns 1 when a valid ref has been added to `list`, 0 otherwise */\n+static int process_ref_v2(const char *line, struct ref ***list)\n+{\n+\tint ret = 1;\n+\tint i = 0;\n+\tstruct object_id old_oid;\n+\tstruct ref *ref;\n+\tstruct string_list line_sections = STRING_LIST_INIT_DUP;\n+\tconst char *end;\n+\n+\t/*\n+\t * Ref lines have a number of fields which are space deliminated. The\n+\t * first field is the OID of the ref. The second field is the ref\n+\t * name. Subsequent fields (symref-target and peeled) are optional and\n+\t * don't have a particular order.\n+\t */\n+\tif (string_list_split(&line_sections, line, ' ', -1) < 2) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tif (parse_oid_hex(line_sections.items[i++].string, &old_oid, &end) ||\n+\t *end) {\n+\t\tret = 0;\n+\t\tgoto out;\n+\t}\n+\n+\tref = alloc_ref(line_sections.items[i++].string);\n+\n+\toidcpy(&ref->old_oid, &old_oid);\n+\t**list = ref;\n+\t*list = &ref->next;\n+\n+\tfor (; i < line_sections.nr; i++) {\n+\t\tconst char *arg = line_sections.items[i].string;\n+\t\tif (skip_prefix(arg, \"symref-target:\", &arg))\n+\t\t\tref->symref = xstrdup(arg);\n+\n+\t\tif (skip_prefix(arg, \"peeled:\", &arg)) {\n+\t\t\tstruct object_id peeled_oid;\n+\t\t\tchar *peeled_name;\n+\t\t\tstruct ref *peeled;\n+\t\t\tif (parse_oid_hex(arg, &peeled_oid, &end) || *end) {\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tpeeled_name = xstrfmt(\"%s^{}\", ref->name);\n+\t\t\tpeeled = alloc_ref(peeled_name);\n+\n+\t\t\toidcpy(&peeled->old_oid, &peeled_oid);\n+\t\t\t**list = peeled;\n+\t\t\t*list = &peeled->next;\n+\n+\t\t\tfree(peeled_name);\n+\t\t}\n+\t}\n+\n+out:\n+\tstring_list_clear(&line_sections, 0);\n+\treturn ret;\n+}\n+\n+struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n+\t\t\t struct ref **list, int for_push,\n+\t\t\t const struct argv_array *ref_patterns)\n+{\n+\tint i;\n+\t*list = NULL;\n+\n+\tif (server_supports_v2(\"ls-refs\", 1))\n+\t\tpacket_write_fmt(fd_out, \"command=ls-refs\\n\");\n+\n+\tif (server_supports_v2(\"agent\", 0))\n+\t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n+\n+\tpacket_delim(fd_out);\n+\t/* When pushing we don't want to request the peeled tags */\n+\tif (!for_push)\n+\t\tpacket_write_fmt(fd_out, \"peel\\n\");\n+\tpacket_write_fmt(fd_out, \"symrefs\\n\");\n+\tfor (i = 0; ref_patterns && i < ref_patterns->argc; i++) {\n+\t\tpacket_write_fmt(fd_out, \"ref-pattern %s\\n\",\n+\t\t\t\t ref_patterns->argv[i]);\n+\t}\n+\tpacket_flush(fd_out);\n+\n+\t/* Process response from server */\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tif (!process_ref_v2(reader->line, &list))\n+\t\t\tdie(\"invalid ls-refs response: %s\", reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH)\n+\t\tdie(\"expected flush after ref listing\");\n+\n+\treturn list;\n+}\n+\n static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)\n {\n \tint len;\n@@ -336,7 +464,7 @@ int parse_feature_request(const char *feature_list, const char *feature)\n \n const char *server_feature_value(const char *feature, int *len)\n {\n-\treturn parse_feature_value(server_capabilities, feature, len);\n+\treturn parse_feature_value(server_capabilities_v1, feature, len);\n }\n \n int server_supports(const char *feature)\ndiff --git a/connect.h b/connect.h\nindex cdb8979dc..8898d4495 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -16,4 +16,6 @@ extern int url_is_local_not_ssh(const char *url);\n struct packet_reader;\n extern enum protocol_version discover_version(struct packet_reader *reader);\n \n+extern int server_supports_v2(const char *c, int die_on_error);\n+\n #endif\ndiff --git a/remote.h b/remote.h\nindex 2016461df..3a9db30cf 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -151,11 +151,17 @@ void free_refs(struct ref *ref);\n \n struct oid_array;\n struct packet_reader;\n+struct argv_array;\n extern struct ref **get_remote_heads(struct packet_reader *reader,\n \t\t\t\t struct ref **list, unsigned int flags,\n \t\t\t\t struct oid_array *extra_have,\n \t\t\t\t struct oid_array *shallow_points);\n \n+/* Used for protocol v2 in order to retrieve refs from a remote */\n+extern struct ref **get_remote_refs(int fd_out, struct packet_reader *reader,\n+\t\t\t\t struct ref **list, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns);\n+\n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\n \ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nnew file mode 100755\nindex 000000000..dc5f813be\n--- /dev/null\n+++ b/t/t5702-protocol-v2.sh\n@@ -0,0 +1,57 @@\n+#!/bin/sh\n+\n+test_description='test git wire-protocol version 2'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+. ./test-lib.sh\n+\n+# Test protocol v2 with 'git://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\ttest_commit -C \"$daemon_parent\" one\n+'\n+\n+test_expect_success 'list refs with git:// using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -c protocol.version=2 \\\n+\t\tls-remote --symref \"$GIT_DAEMON_URL/parent\" >actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"git> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log &&\n+\n+\tgit ls-remote --symref \"$GIT_DAEMON_URL/parent\" >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+stop_git_daemon\n+\n+# Test protocol v2 with 'file://' transport\n+#\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init file_parent &&\n+\ttest_commit -C file_parent one\n+'\n+\n+test_expect_success 'list refs with file:// using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -c protocol.version=2 \\\n+\t\tls-remote --symref \"file://$(pwd)/file_parent\" >actual &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log &&\n+\n+\tgit ls-remote --symref \"file://$(pwd)/file_parent\" >expect &&\n+\ttest_cmp actual expect\n+'\n+\n+test_done\ndiff --git a/transport.c b/transport.c\nindex 83d9dd1df..ffc6b2614 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -204,7 +204,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340655","messageId":"20180228232252.102167-16-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 15/35] transport: convert get_refs_list to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:32Z","receivedAt":"2018-02-28T23:24:11Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert the 'struct transport' virtual function 'get_refs_list()' to\noptionally take an argv_array of ref patterns. When communicating with\na server using protocol v2 these ref patterns can be sent when\nrequesting a listing of their refs allowing the server to filter the\nrefs it sends based on the sent patterns.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 5 +++--\n transport-internal.h | 9 ++++++++-\n transport.c | 16 +++++++++-------\n 3 files changed, 20 insertions(+), 10 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 508015023..4c334b5ee 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1026,7 +1026,8 @@ static int has_attribute(const char *attrs, const char *attr) {\n \t}\n }\n \n-static struct ref *get_refs_list(struct transport *transport, int for_push)\n+static struct ref *get_refs_list(struct transport *transport, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns)\n {\n \tstruct helper_data *data = transport->data;\n \tstruct child_process *helper;\n@@ -1039,7 +1040,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)\n \n \tif (process_connect(transport, for_push)) {\n \t\tdo_take_over(transport);\n-\t\treturn transport->vtable->get_refs_list(transport, for_push);\n+\t\treturn transport->vtable->get_refs_list(transport, for_push, ref_patterns);\n \t}\n \n \tif (data->push && for_push)\ndiff --git a/transport-internal.h b/transport-internal.h\nindex 3c1a29d72..36fcee437 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -3,6 +3,7 @@\n \n struct ref;\n struct transport;\n+struct argv_array;\n \n struct transport_vtable {\n \t/**\n@@ -17,11 +18,17 @@ struct transport_vtable {\n \t * the transport to try to share connections, for_push is a\n \t * hint as to whether the ultimate operation is a push or a fetch.\n \t *\n+\t * If communicating using protocol v2 a list of patterns can be\n+\t * provided to be sent to the server to enable it to limit the ref\n+\t * advertisement. Since ref filtering is done on the server's end,\n+\t * this can return refs which don't match the provided ref_patterns.\n+\t *\n \t * If the transport is able to determine the remote hash for\n \t * the ref without a huge amount of effort, it should store it\n \t * in the ref's old_sha1 field; otherwise it should be all 0.\n \t **/\n-\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push);\n+\tstruct ref *(*get_refs_list)(struct transport *transport, int for_push,\n+\t\t\t\t const struct argv_array *ref_patterns);\n \n \t/**\n \t * Fetch the objects for the given refs. Note that this gets\ndiff --git a/transport.c b/transport.c\nindex ffc6b2614..c54a44630 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -72,7 +72,7 @@ struct bundle_transport_data {\n \tstruct bundle_header header;\n };\n \n-static struct ref *get_refs_from_bundle(struct transport *transport, int for_push)\n+static struct ref *get_refs_from_bundle(struct transport *transport, int for_push, const struct argv_array *ref_patterns)\n {\n \tstruct bundle_transport_data *data = transport->data;\n \tstruct ref *result = NULL;\n@@ -189,7 +189,8 @@ static int connect_setup(struct transport *transport, int for_push)\n \treturn 0;\n }\n \n-static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n+static struct ref *get_refs_via_connect(struct transport *transport, int for_push,\n+\t\t\t\t\tconst struct argv_array *ref_patterns)\n {\n \tstruct git_transport_data *data = transport->data;\n \tstruct ref *refs = NULL;\n@@ -204,7 +205,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tdata->version = discover_version(&reader);\n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push, NULL);\n+\t\tget_remote_refs(data->fd[1], &reader, &refs, for_push,\n+\t\t\t\tref_patterns);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n@@ -250,7 +252,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.update_shallow = data->options.update_shallow;\n \n \tif (!data->got_remote_heads)\n-\t\trefs_tmp = get_refs_via_connect(transport, 0);\n+\t\trefs_tmp = get_refs_via_connect(transport, 0, NULL);\n \n \tswitch (data->version) {\n \tcase protocol_v2:\n@@ -568,7 +570,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \tint ret = 0;\n \n \tif (!data->got_remote_heads)\n-\t\tget_refs_via_connect(transport, 1);\n+\t\tget_refs_via_connect(transport, 1, NULL);\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n@@ -1028,7 +1030,7 @@ int transport_push(struct transport *transport,\n \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n \t\t\treturn -1;\n \n-\t\tremote_refs = transport->vtable->get_refs_list(transport, 1);\n+\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n \n \t\tif (flags & TRANSPORT_PUSH_ALL)\n \t\t\tmatch_flags |= MATCH_REFS_ALL;\n@@ -1137,7 +1139,7 @@ int transport_push(struct transport *transport,\n const struct ref *transport_get_remote_refs(struct transport *transport)\n {\n \tif (!transport->got_remote_refs) {\n-\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0);\n+\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n \t\ttransport->got_remote_refs = 1;\n \t}\n \n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340656","messageId":"20180228232252.102167-14-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 13/35] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:30Z","receivedAt":"2018-02-28T23:24:14Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the ls-refs server command. In protocol v2, the ls-refs\ncommand is used to request the ref advertisement from the server. Since\nit is a command which can be requested (as opposed to mandatory in v1),\na client can sent a number of parameters in its request to limit the ref\nadvertisement based on provided ref-patterns.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/technical/protocol-v2.txt | 36 ++++++\n Makefile | 1 +\n ls-refs.c | 144 ++++++++++++++++++++++++\n ls-refs.h | 9 ++\n serve.c | 8 ++\n t/t5701-git-serve.sh | 115 +++++++++++++++++++\n 6 files changed, 313 insertions(+)\n create mode 100644 ls-refs.c\n create mode 100644 ls-refs.h\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex b02eefc21..7f50e6462 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -169,3 +169,39 @@ printable ASCII characters except space (i.e., the byte range 32 < x <\n \"git/1.8.3.1\"). The agent strings are purely informative for statistics\n and debugging purposes, and MUST NOT be used to programmatically assume\n the presence or absence of particular features.\n+\n+ ls-refs\n+---------\n+\n+`ls-refs` is the command used to request a reference advertisement in v2.\n+Unlike the current reference advertisement, ls-refs takes in arguments\n+which can be used to limit the refs sent from the server.\n+\n+Additional features not supported in the base command will be advertised\n+as the value of the command in the capability advertisement in the form\n+of a space separated list of features, e.g. \"=\n+\".\n+\n+ls-refs takes in the following arguments:\n+\n+ symrefs\n+\tIn addition to the object pointed by it, show the underlying ref\n+\tpointed by it when showing a symbolic ref.\n+ peel\n+\tShow peeled tags.\n+ ref-pattern \n+\tWhen specified, only references matching one of the provided\n+\tpatterns are displayed. A pattern is either a valid refname\n+\t(e.g. refs/heads/master), in which a ref must match the pattern\n+\texactly, or a prefix of a ref followed by a single '*' wildcard\n+\tcharacter (e.g. refs/heads/*), in which a ref must have a prefix\n+\tequal to the pattern up to the wildcard character.\n+\n+The output of ls-refs is as follows:\n+\n+ output = *ref\n+\t flush-pkt\n+ ref = PKT-LINE(obj-id SP refname *(SP ref-attribute) LF)\n+ ref-attribute = (symref | peeled)\n+ symref = \"symref-target:\" symref-target\n+ peeled = \"peeled:\" obj-id\ndiff --git a/Makefile b/Makefile\nindex 18c255428..e50927cfb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -825,6 +825,7 @@ LIB_OBJS += list-objects-filter-options.o\n LIB_OBJS += ll-merge.o\n LIB_OBJS += lockfile.o\n LIB_OBJS += log-tree.o\n+LIB_OBJS += ls-refs.o\n LIB_OBJS += mailinfo.o\n LIB_OBJS += mailmap.o\n LIB_OBJS += match-trees.o\ndiff --git a/ls-refs.c b/ls-refs.c\nnew file mode 100644\nindex 000000000..91d7deb34\n--- /dev/null\n+++ b/ls-refs.c\n@@ -0,0 +1,144 @@\n+#include \"cache.h\"\n+#include \"repository.h\"\n+#include \"refs.h\"\n+#include \"remote.h\"\n+#include \"argv-array.h\"\n+#include \"ls-refs.h\"\n+#include \"pkt-line.h\"\n+\n+struct ref_pattern {\n+\tchar *pattern;\n+\tint wildcard_pos; /* If > 0, indicates the position of the wildcard */\n+};\n+\n+struct pattern_list {\n+\tstruct ref_pattern *patterns;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+static void add_pattern(struct pattern_list *patterns, const char *pattern)\n+{\n+\tstruct ref_pattern p;\n+\tconst char *wildcard;\n+\n+\tp.pattern = strdup(pattern);\n+\n+\twildcard = strchr(pattern, '*');\n+\tif (wildcard) {\n+\t\tp.wildcard_pos = wildcard - pattern;\n+\t} else {\n+\t\tp.wildcard_pos = -1;\n+\t}\n+\n+\tALLOC_GROW(patterns->patterns,\n+\t\t patterns->nr + 1,\n+\t\t patterns->alloc);\n+\tpatterns->patterns[patterns->nr++] = p;\n+}\n+\n+static void clear_patterns(struct pattern_list *patterns)\n+{\n+\tint i;\n+\tfor (i = 0; i < patterns->nr; i++)\n+\t\tfree(patterns->patterns[i].pattern);\n+\tFREE_AND_NULL(patterns->patterns);\n+\tpatterns->nr = 0;\n+\tpatterns->alloc = 0;\n+}\n+\n+/*\n+ * Check if one of the patterns matches the tail part of the ref.\n+ * If no patterns were provided, all refs match.\n+ */\n+static int ref_match(const struct pattern_list *patterns, const char *refname)\n+{\n+\tint i;\n+\n+\tif (!patterns->nr)\n+\t\treturn 1; /* no restriction */\n+\n+\tfor (i = 0; i < patterns->nr; i++) {\n+\t\tconst struct ref_pattern *p = &patterns->patterns[i];\n+\n+\t\t/* No wildcard, exact match expected */\n+\t\tif (p->wildcard_pos < 0) {\n+\t\t\tif (!strcmp(refname, p->pattern))\n+\t\t\t\treturn 1;\n+\t\t} else {\n+\t\t\t/* Wildcard, prefix match until the wildcard */\n+\t\t\tif (!strncmp(refname, p->pattern, p->wildcard_pos))\n+\t\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\treturn 0;\n+}\n+\n+struct ls_refs_data {\n+\tunsigned peel;\n+\tunsigned symrefs;\n+\tstruct pattern_list patterns;\n+};\n+\n+static int send_ref(const char *refname, const struct object_id *oid,\n+\t\t int flag, void *cb_data)\n+{\n+\tstruct ls_refs_data *data = cb_data;\n+\tconst char *refname_nons = strip_namespace(refname);\n+\tstruct strbuf refline = STRBUF_INIT;\n+\n+\tif (!ref_match(&data->patterns, refname))\n+\t\treturn 0;\n+\n+\tstrbuf_addf(&refline, \"%s %s\", oid_to_hex(oid), refname_nons);\n+\tif (data->symrefs && flag & REF_ISSYMREF) {\n+\t\tstruct object_id unused;\n+\t\tconst char *symref_target = resolve_ref_unsafe(refname, 0,\n+\t\t\t\t\t\t\t &unused,\n+\t\t\t\t\t\t\t &flag);\n+\n+\t\tif (!symref_target)\n+\t\t\tdie(\"'%s' is a symref but it is not?\", refname);\n+\n+\t\tstrbuf_addf(&refline, \" symref-target:%s\", symref_target);\n+\t}\n+\n+\tif (data->peel) {\n+\t\tstruct object_id peeled;\n+\t\tif (!peel_ref(refname, &peeled))\n+\t\t\tstrbuf_addf(&refline, \" peeled:%s\", oid_to_hex(&peeled));\n+\t}\n+\n+\tstrbuf_addch(&refline, '\\n');\n+\tpacket_write(1, refline.buf, refline.len);\n+\n+\tstrbuf_release(&refline);\n+\treturn 0;\n+}\n+\n+int ls_refs(struct repository *r, struct argv_array *keys, struct argv_array *args)\n+{\n+\tint i;\n+\tstruct ls_refs_data data;\n+\n+\tmemset(&data, 0, sizeof(data));\n+\n+\tfor (i = 0; i < args->argc; i++) {\n+\t\tconst char *arg = args->argv[i];\n+\t\tconst char *out;\n+\n+\t\tif (!strcmp(\"peel\", arg))\n+\t\t\tdata.peel = 1;\n+\t\telse if (!strcmp(\"symrefs\", arg))\n+\t\t\tdata.symrefs = 1;\n+\t\telse if (skip_prefix(arg, \"ref-pattern \", &out))\n+\t\t\tadd_pattern(&data.patterns, out);\n+\t}\n+\n+\thead_ref_namespaced(send_ref, &data);\n+\tfor_each_namespaced_ref(send_ref, &data);\n+\tpacket_flush(1);\n+\tclear_patterns(&data.patterns);\n+\treturn 0;\n+}\ndiff --git a/ls-refs.h b/ls-refs.h\nnew file mode 100644\nindex 000000000..9e4c57bfe\n--- /dev/null\n+++ b/ls-refs.h\n@@ -0,0 +1,9 @@\n+#ifndef LS_REFS_H\n+#define LS_REFS_H\n+\n+struct repository;\n+struct argv_array;\n+extern int ls_refs(struct repository *r, struct argv_array *keys,\n+\t\t struct argv_array *args);\n+\n+#endif /* LS_REFS_H */\ndiff --git a/serve.c b/serve.c\nindex cf23179b9..c7925c5c7 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -4,8 +4,15 @@\n #include \"pkt-line.h\"\n #include \"version.h\"\n #include \"argv-array.h\"\n+#include \"ls-refs.h\"\n #include \"serve.h\"\n \n+static int always_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\treturn 1;\n+}\n+\n static int agent_advertise(struct repository *r,\n \t\t\t struct strbuf *value)\n {\n@@ -44,6 +51,7 @@ struct protocol_capability {\n \n static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n+\t{ \"ls-refs\", always_advertise, ls_refs },\n };\n \n static void advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex affbad097..11aeb0541 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -8,6 +8,7 @@ test_expect_success 'test capability advertisement' '\n \tcat >expect <<-EOF &&\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n+\tls-refs\n \t0000\n \tEOF\n \n@@ -57,4 +58,118 @@ test_expect_success 'request invalid command' '\n \ttest_i18ngrep \"invalid command\" err\n '\n \n+# Test the basics of ls-refs\n+#\n+test_expect_success 'setup some refs and tags' '\n+\ttest_commit one &&\n+\tgit branch dev master &&\n+\ttest_commit two &&\n+\tgit symbolic-ref refs/heads/release refs/heads/master &&\n+\tgit tag -a -m \"annotated tag\" annotated-tag\n+'\n+\n+test_expect_success 'basics of ls-refs' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse HEAD) HEAD\n+\t$(git rev-parse refs/heads/dev) refs/heads/dev\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/heads/release) refs/heads/release\n+\t$(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag\n+\t$(git rev-parse refs/tags/one) refs/tags/one\n+\t$(git rev-parse refs/tags/two) refs/tags/two\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'basic ref-patterns' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tref-pattern refs/heads/master\n+\tref-pattern refs/tags/one\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/tags/one) refs/tags/one\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'wildcard ref-patterns' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tref-pattern refs/heads/*\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/heads/dev) refs/heads/dev\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/heads/release) refs/heads/release\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'peel parameter' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tpeel\n+\tref-pattern refs/tags/*\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/tags/annotated-tag) refs/tags/annotated-tag peeled:$(git rev-parse refs/tags/annotated-tag^{})\n+\t$(git rev-parse refs/tags/one) refs/tags/one\n+\t$(git rev-parse refs/tags/two) refs/tags/two\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n+test_expect_success 'symrefs parameter' '\n+\ttest-pkt-line pack >in <<-EOF &&\n+\tcommand=ls-refs\n+\t0001\n+\tsymrefs\n+\tref-pattern refs/heads/*\n+\t0000\n+\tEOF\n+\n+\tcat >expect <<-EOF &&\n+\t$(git rev-parse refs/heads/dev) refs/heads/dev\n+\t$(git rev-parse refs/heads/master) refs/heads/master\n+\t$(git rev-parse refs/heads/release) refs/heads/release symref-target:refs/heads/master\n+\t0000\n+\tEOF\n+\n+\tgit serve --stateless-rpc out &&\n+\ttest-pkt-line unpack actual &&\n+\ttest_cmp actual expect\n+'\n+\n test_done\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340657","messageId":"20180228232252.102167-20-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 19/35] push: pass ref patterns when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:36Z","receivedAt":"2018-02-28T23:24:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct a list of ref patterns to be passed to 'get_refs_list()' from\nthe refspec to be used during the push. This list of ref patterns will\nbe used to allow the server to filter the ref advertisement when\ncommunicating using protocol v2.\n\nSigned-off-by: Brandon Williams \n---\n transport.c | 26 +++++++++++++++++++++++++-\n 1 file changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/transport.c b/transport.c\nindex dfc603b36..bf7ba6879 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1026,11 +1026,35 @@ int transport_push(struct transport *transport,\n \t\tint porcelain = flags & TRANSPORT_PUSH_PORCELAIN;\n \t\tint pretend = flags & TRANSPORT_PUSH_DRY_RUN;\n \t\tint push_ret, ret, err;\n+\t\tstruct refspec *tmp_rs;\n+\t\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n+\t\tint i;\n \n \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n \t\t\treturn -1;\n \n-\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n+\t\ttmp_rs = parse_push_refspec(refspec_nr, refspec);\n+\t\tfor (i = 0; i < refspec_nr; i++) {\n+\t\t\tconst char *pattern = NULL;\n+\n+\t\t\tif (tmp_rs[i].dst)\n+\t\t\t\tpattern = tmp_rs[i].dst;\n+\t\t\telse if (tmp_rs[i].src && !tmp_rs[i].exact_sha1)\n+\t\t\t\tpattern = tmp_rs[i].src;\n+\n+\t\t\tif (pattern) {\n+\t\t\t\tif (tmp_rs[i].pattern)\n+\t\t\t\t\targv_array_push(&ref_patterns, pattern);\n+\t\t\t\telse\n+\t\t\t\t\texpand_ref_pattern(&ref_patterns, pattern);\n+\t\t\t}\n+\t\t}\n+\n+\t\tremote_refs = transport->vtable->get_refs_list(transport, 1,\n+\t\t\t\t\t\t\t &ref_patterns);\n+\n+\t\targv_array_clear(&ref_patterns);\n+\t\tfree_refspec(refspec_nr, tmp_rs);\n \n \t\tif (flags & TRANSPORT_PUSH_ALL)\n \t\t\tmatch_flags |= MATCH_REFS_ALL;\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340658","messageId":"20180228232252.102167-21-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 20/35] upload-pack: introduce fetch server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:37Z","receivedAt":"2018-02-28T23:24:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the 'fetch' server command.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/technical/protocol-v2.txt | 128 ++++++++++++\n serve.c | 2 +\n t/t5701-git-serve.sh | 1 +\n upload-pack.c | 267 ++++++++++++++++++++++++\n upload-pack.h | 5 +\n 5 files changed, 403 insertions(+)\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 7f50e6462..99c70a1e4 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -205,3 +205,131 @@ The output of ls-refs is as follows:\n ref-attribute = (symref | peeled)\n symref = \"symref-target:\" symref-target\n peeled = \"peeled:\" obj-id\n+\n+ fetch\n+-------\n+\n+`fetch` is the command used to fetch a packfile in v2. It can be looked\n+at as a modified version of the v1 fetch where the ref-advertisement is\n+stripped out (since the `ls-refs` command fills that role) and the\n+message format is tweaked to eliminate redundancies and permit easy\n+addition of future extensions.\n+\n+Additional features not supported in the base command will be advertised\n+as the value of the command in the capability advertisement in the form\n+of a space separated list of features, e.g. \"=\n+\".\n+\n+A `fetch` request can take the following arguments:\n+\n+ want \n+\tIndicates to the server an object which the client wants to\n+\tretrieve. Wants can be anything and are not limited to\n+\tadvertised objects.\n+\n+ have \n+\tIndicates to the server an object which the client has locally.\n+\tThis allows the server to make a packfile which only contains\n+\tthe objects that the client needs. Multiple 'have' lines can be\n+\tsupplied.\n+\n+ done\n+\tIndicates to the server that negotiation should terminate (or\n+\tnot even begin if performing a clone) and that the server should\n+\tuse the information supplied in the request to construct the\n+\tpackfile.\n+\n+ thin-pack\n+\tRequest that a thin pack be sent, which is a pack with deltas\n+\twhich reference base objects not contained within the pack (but\n+\tare known to exist at the receiving end). This can reduce the\n+\tnetwork traffic significantly, but it requires the receiving end\n+\tto know how to \"thicken\" these packs by adding the missing bases\n+\tto the pack.\n+\n+ no-progress\n+\tRequest that progress information that would normally be sent on\n+\tside-band channel 2, during the packfile transfer, should not be\n+\tsent. However, the side-band channel 3 is still used for error\n+\tresponses.\n+\n+ include-tag\n+\tRequest that annotated tags should be sent if the objects they\n+\tpoint to are being sent.\n+\n+ ofs-delta\n+\tIndicate that the client understands PACKv2 with delta referring\n+\tto its base by position in pack rather than by an oid. That is,\n+\tthey can read OBJ_OFS_DELTA (ake type 6) in a packfile.\n+\n+The response of `fetch` is broken into a number of sections separated by\n+delimiter packets (0001), with each section beginning with its section\n+header.\n+\n+ output = *section\n+ section = (acknowledgments | packfile)\n+\t (flush-pkt | delim-pkt)\n+\n+ acknowledgments = PKT-LINE(\"acknowledgments\" LF)\n+\t\t (nak | *ack)\n+\t\t (ready)\n+ ready = PKT-LINE(\"ready\" LF)\n+ nak = PKT-LINE(\"NAK\" LF)\n+ ack = PKT-LINE(\"ACK\" SP obj-id LF)\n+\n+ packfile = PKT-LINE(\"packfile\" LF)\n+\t [PACKFILE]\n+\n+----\n+ acknowledgments section\n+\t* Always begins with the section header \"acknowledgments\"\n+\n+\t* The server will respond with \"NAK\" if none of the object ids sent\n+\t as have lines were common.\n+\n+\t* The server will respond with \"ACK obj-id\" for all of the\n+\t object ids sent as have lines which are common.\n+\n+\t* A response cannot have both \"ACK\" lines as well as a \"NAK\"\n+\t line.\n+\n+\t* The server will respond with a \"ready\" line indicating that\n+\t the server has found an acceptable common base and is ready to\n+\t make and send a packfile (which will be found in the packfile\n+\t section of the same response)\n+\n+\t* If the client determines that it is finished with negotiations\n+\t by sending a \"done\" line, the acknowledgments sections MUST be\n+\t omitted from the server's response.\n+\n+\t* If the server has found a suitable cut point and has decided\n+\t to send a \"ready\" line, then the server can decide to (as an\n+\t optimization) omit any \"ACK\" lines it would have sent during\n+\t its response. This is because the server will have already\n+\t determined the objects it plans to send to the client and no\n+\t further negotiation is needed.\n+\n+----\n+ packfile section\n+\t* Always begins with the section header \"packfile\"\n+\n+\t* The transmission of the packfile begins immediately after the\n+\t section header\n+\n+\t* The data transfer of the packfile is always multiplexed, using\n+\t the same semantics of the 'side-band-64k' capability from\n+\t protocol version 1. This means that each packet, during the\n+\t packfile data stream, is made up of a leading 4-byte pkt-line\n+\t length (typical of the pkt-line format), followed by a 1-byte\n+\t stream code, followed by the actual data.\n+\n+\t The stream code can be one of:\n+\t\t1 - pack data\n+\t\t2 - progress messages\n+\t\t3 - fatal error message just before stream aborts\n+\n+\t* This section is only included if the client has sent 'want'\n+\t lines in its request and either requested that no more\n+\t negotiation be done by sending 'done' or if the server has\n+\t decided it has found a sufficient cut point to produce a\n+\t packfile.\ndiff --git a/serve.c b/serve.c\nindex c7925c5c7..05cc434cf 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -6,6 +6,7 @@\n #include \"argv-array.h\"\n #include \"ls-refs.h\"\n #include \"serve.h\"\n+#include \"upload-pack.h\"\n \n static int always_advertise(struct repository *r,\n \t\t\t struct strbuf *value)\n@@ -52,6 +53,7 @@ struct protocol_capability {\n static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n \t{ \"ls-refs\", always_advertise, ls_refs },\n+\t{ \"fetch\", always_advertise, upload_pack_v2 },\n };\n \n static void advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex 11aeb0541..cc5918a67 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -9,6 +9,7 @@ test_expect_success 'test capability advertisement' '\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs\n+\tfetch\n \t0000\n \tEOF\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex 1e8a9e1ca..2af6b1382 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -18,6 +18,7 @@\n #include \"prio-queue.h\"\n #include \"protocol.h\"\n #include \"upload-pack.h\"\n+#include \"serve.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -1065,3 +1066,269 @@ void upload_pack(struct upload_pack_options *options)\n \t\tcreate_pack_file();\n \t}\n }\n+\n+struct upload_pack_data {\n+\tstruct object_array wants;\n+\tstruct oid_array haves;\n+\n+\tunsigned stateless_rpc : 1;\n+\n+\tunsigned use_thin_pack : 1;\n+\tunsigned use_ofs_delta : 1;\n+\tunsigned no_progress : 1;\n+\tunsigned use_include_tag : 1;\n+\tunsigned done : 1;\n+};\n+\n+static void upload_pack_data_init(struct upload_pack_data *data)\n+{\n+\tstruct object_array wants = OBJECT_ARRAY_INIT;\n+\tstruct oid_array haves = OID_ARRAY_INIT;\n+\n+\tmemset(data, 0, sizeof(*data));\n+\tdata->wants = wants;\n+\tdata->haves = haves;\n+}\n+\n+static void upload_pack_data_clear(struct upload_pack_data *data)\n+{\n+\tobject_array_clear(&data->wants);\n+\toid_array_clear(&data->haves);\n+}\n+\n+static int parse_want(const char *line)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"want \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tstruct object *o;\n+\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"git upload-pack: protocol error, \"\n+\t\t\t \"expected to get oid, not '%s'\", line);\n+\n+\t\to = parse_object(&oid);\n+\t\tif (!o) {\n+\t\t\tpacket_write_fmt(1,\n+\t\t\t\t\t \"ERR upload-pack: not our ref %s\",\n+\t\t\t\t\t oid_to_hex(&oid));\n+\t\t\tdie(\"git upload-pack: not our ref %s\",\n+\t\t\t oid_to_hex(&oid));\n+\t\t}\n+\n+\t\tif (!(o->flags & WANTED)) {\n+\t\t\to->flags |= WANTED;\n+\t\t\tadd_object_array(o, NULL, &want_obj);\n+\t\t}\n+\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int parse_have(const char *line, struct oid_array *haves)\n+{\n+\tconst char *arg;\n+\tif (skip_prefix(line, \"have \", &arg)) {\n+\t\tstruct object_id oid;\n+\n+\t\tif (get_oid_hex(arg, &oid))\n+\t\t\tdie(\"git upload-pack: expected SHA1 object, got '%s'\", arg);\n+\t\toid_array_append(haves, &oid);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void process_args(struct argv_array *args, struct upload_pack_data *data)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < args->argc; i++) {\n+\t\tconst char *arg = args->argv[i];\n+\n+\t\t/* process want */\n+\t\tif (parse_want(arg))\n+\t\t\tcontinue;\n+\t\t/* process have line */\n+\t\tif (parse_have(arg, &data->haves))\n+\t\t\tcontinue;\n+\n+\t\t/* process args like thin-pack */\n+\t\tif (!strcmp(arg, \"thin-pack\")) {\n+\t\t\tuse_thin_pack = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"ofs-delta\")) {\n+\t\t\tuse_ofs_delta = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"no-progress\")) {\n+\t\t\tno_progress = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"include-tag\")) {\n+\t\t\tuse_include_tag = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"done\")) {\n+\t\t\tdata->done = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* ignore unknown lines maybe? */\n+\t\tdie(\"unexpect line: '%s'\", arg);\n+\t}\n+}\n+\n+static int process_haves(struct oid_array *haves, struct oid_array *common)\n+{\n+\tint i;\n+\n+\t/* Process haves */\n+\tfor (i = 0; i < haves->nr; i++) {\n+\t\tconst struct object_id *oid = &haves->oid[i];\n+\t\tstruct object *o;\n+\t\tint we_knew_they_have = 0;\n+\n+\t\tif (!has_object_file(oid))\n+\t\t\tcontinue;\n+\n+\t\toid_array_append(common, oid);\n+\n+\t\to = parse_object(oid);\n+\t\tif (!o)\n+\t\t\tdie(\"oops (%s)\", oid_to_hex(oid));\n+\t\tif (o->type == OBJ_COMMIT) {\n+\t\t\tstruct commit_list *parents;\n+\t\t\tstruct commit *commit = (struct commit *)o;\n+\t\t\tif (o->flags & THEY_HAVE)\n+\t\t\t\twe_knew_they_have = 1;\n+\t\t\telse\n+\t\t\t\to->flags |= THEY_HAVE;\n+\t\t\tif (!oldest_have || (commit->date < oldest_have))\n+\t\t\t\toldest_have = commit->date;\n+\t\t\tfor (parents = commit->parents;\n+\t\t\t parents;\n+\t\t\t parents = parents->next)\n+\t\t\t\tparents->item->object.flags |= THEY_HAVE;\n+\t\t}\n+\t\tif (!we_knew_they_have)\n+\t\t\tadd_object_array(o, NULL, &have_obj);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int send_acks(struct oid_array *acks, struct strbuf *response)\n+{\n+\tint i;\n+\n+\tpacket_buf_write(response, \"acknowledgments\\n\");\n+\n+\t/* Send Acks */\n+\tif (!acks->nr)\n+\t\tpacket_buf_write(response, \"NAK\\n\");\n+\n+\tfor (i = 0; i < acks->nr; i++) {\n+\t\tpacket_buf_write(response, \"ACK %s\\n\",\n+\t\t\t\t oid_to_hex(&acks->oid[i]));\n+\t}\n+\n+\tif (ok_to_give_up()) {\n+\t\t/* Send Ready */\n+\t\tpacket_buf_write(response, \"ready\\n\");\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int process_haves_and_send_acks(struct upload_pack_data *data)\n+{\n+\tstruct oid_array common = OID_ARRAY_INIT;\n+\tstruct strbuf response = STRBUF_INIT;\n+\tint ret = 0;\n+\n+\tprocess_haves(&data->haves, &common);\n+\tif (data->done) {\n+\t\tret = 1;\n+\t} else if (send_acks(&common, &response)) {\n+\t\tpacket_buf_delim(&response);\n+\t\tret = 1;\n+\t} else {\n+\t\t/* Add Flush */\n+\t\tpacket_buf_flush(&response);\n+\t\tret = 0;\n+\t}\n+\n+\t/* Send response */\n+\twrite_or_die(1, response.buf, response.len);\n+\tstrbuf_release(&response);\n+\n+\toid_array_clear(&data->haves);\n+\toid_array_clear(&common);\n+\treturn ret;\n+}\n+\n+enum fetch_state {\n+\tFETCH_PROCESS_ARGS = 0,\n+\tFETCH_SEND_ACKS,\n+\tFETCH_SEND_PACK,\n+\tFETCH_DONE,\n+};\n+\n+int upload_pack_v2(struct repository *r, struct argv_array *keys,\n+\t\t struct argv_array *args)\n+{\n+\tenum fetch_state state = FETCH_PROCESS_ARGS;\n+\tstruct upload_pack_data data;\n+\n+\tupload_pack_data_init(&data);\n+\tuse_sideband = LARGE_PACKET_MAX;\n+\n+\twhile (state != FETCH_DONE) {\n+\t\tswitch (state) {\n+\t\tcase FETCH_PROCESS_ARGS:\n+\t\t\tprocess_args(args, &data);\n+\n+\t\t\tif (!want_obj.nr) {\n+\t\t\t\t/*\n+\t\t\t\t * Request didn't contain any 'want' lines,\n+\t\t\t\t * guess they didn't want anything.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\t} else if (data.haves.nr) {\n+\t\t\t\t/*\n+\t\t\t\t * Request had 'have' lines, so lets ACK them.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_SEND_ACKS;\n+\t\t\t} else {\n+\t\t\t\t/*\n+\t\t\t\t * Request had 'want's but no 'have's so we can\n+\t\t\t\t * immedietly go to construct and send a pack.\n+\t\t\t\t */\n+\t\t\t\tstate = FETCH_SEND_PACK;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_ACKS:\n+\t\t\tif (process_haves_and_send_acks(&data))\n+\t\t\t\tstate = FETCH_SEND_PACK;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_PACK:\n+\t\t\tpacket_write_fmt(1, \"packfile\\n\");\n+\t\t\tcreate_pack_file();\n+\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_DONE:\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n+\tupload_pack_data_clear(&data);\n+\treturn 0;\n+}\ndiff --git a/upload-pack.h b/upload-pack.h\nindex a71e4dc7e..6b7890238 100644\n--- a/upload-pack.h\n+++ b/upload-pack.h\n@@ -10,4 +10,9 @@ struct upload_pack_options {\n \n void upload_pack(struct upload_pack_options *options);\n \n+struct repository;\n+struct argv_array;\n+extern int upload_pack_v2(struct repository *r, struct argv_array *keys,\n+\t\t\t struct argv_array *args);\n+\n #endif /* UPLOAD_PACK_H */\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340659","messageId":"20180228232252.102167-22-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 21/35] fetch-pack: perform a fetch using v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:38Z","receivedAt":"2018-02-28T23:24:26Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"When communicating with a v2 server, perform a fetch by requesting the\n'fetch' command.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/technical/protocol-v2.txt | 68 +++++-\n builtin/fetch-pack.c | 2 +-\n fetch-pack.c | 270 +++++++++++++++++++++++-\n fetch-pack.h | 4 +-\n serve.c | 2 +-\n t/t5701-git-serve.sh | 2 +-\n t/t5702-protocol-v2.sh | 97 +++++++++\n transport.c | 7 +-\n upload-pack.c | 141 ++++++++++---\n upload-pack.h | 3 +\n 10 files changed, 548 insertions(+), 48 deletions(-)\n\ndiff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt\nindex 99c70a1e4..0d63456fc 100644\n--- a/Documentation/technical/protocol-v2.txt\n+++ b/Documentation/technical/protocol-v2.txt\n@@ -262,12 +262,43 @@ A `fetch` request can take the following arguments:\n \tto its base by position in pack rather than by an oid. That is,\n \tthey can read OBJ_OFS_DELTA (ake type 6) in a packfile.\n \n+ shallow \n+\tA client must notify the server of all commits for which it only\n+\thas shallow copies (meaning that it doesn't have the parents of\n+\ta commit) by supplying a 'shallow ' line for each such\n+\tobject so that the server is aware of the limitations of the\n+\tclient's history. This is so that the server is aware that the\n+\tclient may not have all objects reachable from such commits.\n+\n+ deepen \n+\tRequests that the fetch/clone should be shallow having a commit\n+\tdepth of relative to the remote side.\n+\n+ deepen-relative\n+\tRequests that the semantics of the \"deepen\" command be changed\n+\tto indicate that the depth requested is relative to the client's\n+\tcurrent shallow boundary, instead of relative to the requested\n+\tcommits.\n+\n+ deepen-since \n+\tRequests that the shallow clone/fetch should be cut at a\n+\tspecific time, instead of depth. Internally it's equivalent to\n+\tdoing \"git rev-list --max-age=\". Cannot be used with\n+\t\"deepen\".\n+\n+ deepen-not \n+\tRequests that the shallow clone/fetch should be cut at a\n+\tspecific revision specified by '', instead of a depth.\n+\tInternally it's equivalent of doing \"git rev-list --not \".\n+\tCannot be used with \"deepen\", but can be used with\n+\t\"deepen-since\".\n+\n The response of `fetch` is broken into a number of sections separated by\n delimiter packets (0001), with each section beginning with its section\n header.\n \n output = *section\n- section = (acknowledgments | packfile)\n+ section = (acknowledgments | shallow-info | packfile)\n \t (flush-pkt | delim-pkt)\n \n acknowledgments = PKT-LINE(\"acknowledgments\" LF)\n@@ -277,6 +308,11 @@ header.\n nak = PKT-LINE(\"NAK\" LF)\n ack = PKT-LINE(\"ACK\" SP obj-id LF)\n \n+ shallow-info = PKT-LINE(\"shallow-info\" LF)\n+\t\t *PKT-LINE((shallow | unshallow) LF)\n+ shallow = \"shallow\" SP obj-id\n+ unshallow = \"unshallow\" SP obj-id\n+\n packfile = PKT-LINE(\"packfile\" LF)\n \t [PACKFILE]\n \n@@ -309,6 +345,36 @@ header.\n \t determined the objects it plans to send to the client and no\n \t further negotiation is needed.\n \n+----\n+ shallow-info section\n+\tIf the client has requested a shallow fetch/clone, a shallow\n+\tclient requests a fetch or the server is shallow then the\n+\tserver's response may include a shallow-info section. The\n+\tshallow-info section will be included if (due to one of the\n+\tabove conditions) the server needs to inform the client of any\n+\tshallow boundaries or adjustments to the clients already\n+\texisting shallow boundaries.\n+\n+\t* Always begins with the section header \"shallow-info\"\n+\n+\t* If a positive depth is requested, the server will compute the\n+\t set of commits which are no deeper than the desired depth.\n+\n+\t* The server sends a \"shallow obj-id\" line for each commit whose\n+\t parents will not be sent in the following packfile.\n+\n+\t* The server sends an \"unshallow obj-id\" line for each commit\n+\t which the client has indicated is shallow, but is no longer\n+\t shallow as a result of the fetch (due to its parents being\n+\t sent in the following packfile).\n+\n+\t* The server MUST NOT send any \"unshallow\" lines for anything\n+\t which the client has not indicated was shallow as a part of\n+\t its request.\n+\n+\t* This section is only included if a packfile section is also\n+\t included in the response.\n+\n ----\n packfile section\n \t* Always begins with the section header \"packfile\"\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex b2374ddbb..f9d7d0b5a 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -212,7 +212,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n-\t\t\t &shallow, pack_lockfile_ptr);\n+\t\t\t &shallow, pack_lockfile_ptr, protocol_v0);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 9f6b07ad9..dffcfd66a 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -303,9 +303,9 @@ static void insert_one_alternate_object(struct object *obj)\n #define PIPESAFE_FLUSH 32\n #define LARGE_FLUSH 16384\n \n-static int next_flush(struct fetch_pack_args *args, int count)\n+static int next_flush(int stateless_rpc, int count)\n {\n-\tif (args->stateless_rpc) {\n+\tif (stateless_rpc) {\n \t\tif (count < LARGE_FLUSH)\n \t\t\tcount <<= 1;\n \t\telse\n@@ -461,7 +461,7 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tsend_request(args, fd[1], &req_buf);\n \t\t\tstrbuf_setlen(&req_buf, state_len);\n \t\t\tflushes++;\n-\t\t\tflush_at = next_flush(args, count);\n+\t\t\tflush_at = next_flush(args->stateless_rpc, count);\n \n \t\t\t/*\n \t\t\t * We keep one window \"ahead\" of the other side, and\n@@ -1008,6 +1008,259 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \treturn ref;\n }\n \n+static void add_wants(const struct ref *wants, struct strbuf *req_buf)\n+{\n+\tfor ( ; wants ; wants = wants->next) {\n+\t\tconst struct object_id *remote = &wants->old_oid;\n+\t\tconst char *remote_hex;\n+\t\tstruct object *o;\n+\n+\t\t/*\n+\t\t * If that object is complete (i.e. it is an ancestor of a\n+\t\t * local ref), we tell them we have it but do not have to\n+\t\t * tell them about its ancestors, which they already know\n+\t\t * about.\n+\t\t *\n+\t\t * We use lookup_object here because we are only\n+\t\t * interested in the case we *know* the object is\n+\t\t * reachable and we have already scanned it.\n+\t\t */\n+\t\tif (((o = lookup_object(remote->hash)) != NULL) &&\n+\t\t (o->flags & COMPLETE)) {\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tremote_hex = oid_to_hex(remote);\n+\t\tpacket_buf_write(req_buf, \"want %s\\n\", remote_hex);\n+\t}\n+}\n+\n+static void add_common(struct strbuf *req_buf, struct oidset *common)\n+{\n+\tstruct oidset_iter iter;\n+\tconst struct object_id *oid;\n+\toidset_iter_init(common, &iter);\n+\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n+\t}\n+}\n+\n+static int add_haves(struct strbuf *req_buf, int *haves_to_send, int *in_vain)\n+{\n+\tint ret = 0;\n+\tint haves_added = 0;\n+\tconst struct object_id *oid;\n+\n+\twhile ((oid = get_rev())) {\n+\t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n+\t\tif (++haves_added >= *haves_to_send)\n+\t\t\tbreak;\n+\t}\n+\n+\t*in_vain += haves_added;\n+\tif (!haves_added || *in_vain >= MAX_IN_VAIN) {\n+\t\t/* Send Done */\n+\t\tpacket_buf_write(req_buf, \"done\\n\");\n+\t\tret = 1;\n+\t}\n+\n+\t/* Increase haves to send on next round */\n+\t*haves_to_send = next_flush(1, *haves_to_send);\n+\n+\treturn ret;\n+}\n+\n+static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n+\t\t\t const struct ref *wants, struct oidset *common,\n+\t\t\t int *haves_to_send, int *in_vain)\n+{\n+\tint ret = 0;\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\n+\tif (server_supports_v2(\"fetch\", 1))\n+\t\tpacket_buf_write(&req_buf, \"command=fetch\");\n+\tif (server_supports_v2(\"agent\", 0))\n+\t\tpacket_buf_write(&req_buf, \"agent=%s\", git_user_agent_sanitized());\n+\n+\tpacket_buf_delim(&req_buf);\n+\tif (args->use_thin_pack)\n+\t\tpacket_buf_write(&req_buf, \"thin-pack\");\n+\tif (args->no_progress)\n+\t\tpacket_buf_write(&req_buf, \"no-progress\");\n+\tif (args->include_tag)\n+\t\tpacket_buf_write(&req_buf, \"include-tag\");\n+\tif (prefer_ofs_delta)\n+\t\tpacket_buf_write(&req_buf, \"ofs-delta\");\n+\n+\t/* add wants */\n+\tadd_wants(wants, &req_buf);\n+\n+\t/* Add all of the common commits we've found in previous rounds */\n+\tadd_common(&req_buf, common);\n+\n+\t/* Add initial haves */\n+\tret = add_haves(&req_buf, haves_to_send, in_vain);\n+\n+\t/* Send request */\n+\tpacket_buf_flush(&req_buf);\n+\twrite_or_die(fd_out, req_buf.buf, req_buf.len);\n+\n+\tstrbuf_release(&req_buf);\n+\treturn ret;\n+}\n+\n+/*\n+ * Processes a section header in a server's response and checks if it matches\n+ * `section`. If the value of `peek` is 1, the header line will be peeked (and\n+ * not consumed); if 0, the line will be consumed and the function will die if\n+ * the section header doesn't match what was expected.\n+ */\n+static int process_section_header(struct packet_reader *reader,\n+\t\t\t\t const char *section, int peek)\n+{\n+\tint ret;\n+\n+\tif (packet_reader_peek(reader) != PACKET_READ_NORMAL)\n+\t\tdie(\"error reading packet\");\n+\n+\tret = !strcmp(reader->line, section);\n+\n+\tif (!peek) {\n+\t\tif (!ret)\n+\t\t\tdie(\"expected '%s', received '%s'\",\n+\t\t\t section, reader->line);\n+\t\tpacket_reader_read(reader);\n+\t}\n+\n+\treturn ret;\n+}\n+\n+static int process_acks(struct packet_reader *reader, struct oidset *common)\n+{\n+\t/* received */\n+\tint received_ready = 0;\n+\tint received_ack = 0;\n+\n+\tprocess_section_header(reader, \"acknowledgments\", 0);\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *arg;\n+\n+\t\tif (!strcmp(reader->line, \"NAK\"))\n+\t\t\tcontinue;\n+\n+\t\tif (skip_prefix(reader->line, \"ACK \", &arg)) {\n+\t\t\tstruct object_id oid;\n+\t\t\tif (!get_oid_hex(arg, &oid)) {\n+\t\t\t\tstruct commit *commit;\n+\t\t\t\toidset_insert(common, &oid);\n+\t\t\t\tcommit = lookup_commit(&oid);\n+\t\t\t\tmark_common(commit, 0, 1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!strcmp(reader->line, \"ready\")) {\n+\t\t\tclear_prio_queue(&rev_list);\n+\t\t\treceived_ready = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tdie(\"unexpected acknowledgment line: '%s'\", reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH &&\n+\t reader->status != PACKET_READ_DELIM)\n+\t\tdie(\"error processing acks: %d\", reader->status);\n+\n+\t/* return 0 if no common, 1 if there are common, or 2 if ready */\n+\treturn received_ready ? 2 : (received_ack ? 1 : 0);\n+}\n+\n+enum fetch_state {\n+\tFETCH_CHECK_LOCAL = 0,\n+\tFETCH_SEND_REQUEST,\n+\tFETCH_PROCESS_ACKS,\n+\tFETCH_GET_PACK,\n+\tFETCH_DONE,\n+};\n+\n+static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n+\t\t\t\t int fd[2],\n+\t\t\t\t const struct ref *orig_ref,\n+\t\t\t\t struct ref **sought, int nr_sought,\n+\t\t\t\t char **pack_lockfile)\n+{\n+\tstruct ref *ref = copy_ref_list(orig_ref);\n+\tenum fetch_state state = FETCH_CHECK_LOCAL;\n+\tstruct oidset common = OIDSET_INIT;\n+\tstruct packet_reader reader;\n+\tint in_vain = 0;\n+\tint haves_to_send = INITIAL_FLUSH;\n+\tpacket_reader_init(&reader, fd[0], NULL, 0,\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\n+\twhile (state != FETCH_DONE) {\n+\t\tswitch (state) {\n+\t\tcase FETCH_CHECK_LOCAL:\n+\t\t\tsort_ref_list(&ref, ref_compare_name);\n+\t\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n+\n+\t\t\t/* v2 supports these by default */\n+\t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n+\t\t\tuse_sideband = 2;\n+\n+\t\t\tif (marked)\n+\t\t\t\tfor_each_ref(clear_marks, NULL);\n+\t\t\tmarked = 1;\n+\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n+\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n+\n+\t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n+\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n+\t\t\t\tstate = FETCH_DONE;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_SEND_REQUEST;\n+\t\t\tbreak;\n+\t\tcase FETCH_SEND_REQUEST:\n+\t\t\tif (send_fetch_request(fd[1], args, ref, &common,\n+\t\t\t\t\t &haves_to_send, &in_vain))\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\telse\n+\t\t\t\tstate = FETCH_PROCESS_ACKS;\n+\t\t\tbreak;\n+\t\tcase FETCH_PROCESS_ACKS:\n+\t\t\t/* Process ACKs/NAKs */\n+\t\t\tswitch (process_acks(&reader, &common)) {\n+\t\t\tcase 2:\n+\t\t\t\tstate = FETCH_GET_PACK;\n+\t\t\t\tbreak;\n+\t\t\tcase 1:\n+\t\t\t\tin_vain = 0;\n+\t\t\t\t/* fallthrough */\n+\t\t\tdefault:\n+\t\t\t\tstate = FETCH_SEND_REQUEST;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase FETCH_GET_PACK:\n+\t\t\t/* get the pack */\n+\t\t\tprocess_section_header(&reader, \"packfile\", 0);\n+\t\t\tif (get_pack(args, fd, pack_lockfile))\n+\t\t\t\tdie(_(\"git fetch-pack: fetch failed.\"));\n+\n+\t\t\tstate = FETCH_DONE;\n+\t\t\tbreak;\n+\t\tcase FETCH_DONE:\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n+\toidset_clear(&common);\n+\treturn ref;\n+}\n+\n static void fetch_pack_config(void)\n {\n \tgit_config_get_int(\"fetch.unpacklimit\", &fetch_unpack_limit);\n@@ -1153,7 +1406,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const char *dest,\n \t\t struct ref **sought, int nr_sought,\n \t\t struct oid_array *shallow,\n-\t\t char **pack_lockfile)\n+\t\t char **pack_lockfile,\n+\t\t enum protocol_version version)\n {\n \tstruct ref *ref_cpy;\n \tstruct shallow_info si;\n@@ -1167,8 +1421,12 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tdie(_(\"no matching remote head\"));\n \t}\n \tprepare_shallow_info(&si, shallow);\n-\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n-\t\t\t\t&si, pack_lockfile);\n+\tif (version == protocol_v2)\n+\t\tref_cpy = do_fetch_pack_v2(args, fd, ref, sought, nr_sought,\n+\t\t\t\t\t pack_lockfile);\n+\telse\n+\t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n+\t\t\t\t\t&si, pack_lockfile);\n \treprepare_packed_git();\n \tupdate_shallow(args, sought, nr_sought, &si);\n \tclear_shallow_info(&si);\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex b6aeb43a8..7afca7305 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -3,6 +3,7 @@\n \n #include \"string-list.h\"\n #include \"run-command.h\"\n+#include \"protocol.h\"\n \n struct oid_array;\n \n@@ -43,7 +44,8 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t struct ref **sought,\n \t\t int nr_sought,\n \t\t struct oid_array *shallow,\n-\t\t char **pack_lockfile);\n+\t\t char **pack_lockfile,\n+\t\t enum protocol_version version);\n \n /*\n * Print an appropriate error message for each sought ref that wasn't\ndiff --git a/serve.c b/serve.c\nindex 05cc434cf..c3e58c1e7 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -53,7 +53,7 @@ struct protocol_capability {\n static struct protocol_capability capabilities[] = {\n \t{ \"agent\", agent_advertise, NULL },\n \t{ \"ls-refs\", always_advertise, ls_refs },\n-\t{ \"fetch\", always_advertise, upload_pack_v2 },\n+\t{ \"fetch\", upload_pack_advertise, upload_pack_v2 },\n };\n \n static void advertise_capabilities(void)\ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex cc5918a67..569922f7a 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -9,7 +9,7 @@ test_expect_success 'test capability advertisement' '\n \tversion 2\n \tagent=git/$(git version | cut -d\" \" -f3)\n \tls-refs\n-\tfetch\n+\tfetch=shallow\n \t0000\n \tEOF\n \ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 562610fd2..4365ac273 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -45,6 +45,56 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol\n \ttest_cmp actual expect\n '\n \n+test_expect_success 'clone with git:// using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -c protocol.version=2 \\\n+\t\tclone \"$GIT_DAEMON_URL/parent\" daemon_child &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"clone> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"clone< version 2\" log\n+'\n+\n+test_expect_success 'fetch with git:// using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_commit -C \"$daemon_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -C daemon_child -c protocol.version=2 \\\n+\t\tfetch &&\n+\n+\tgit -C daemon_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"fetch< version 2\" log\n+'\n+\n+test_expect_success 'pull with git:// using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -C daemon_child -c protocol.version=2 \\\n+\t\tpull &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"fetch< version 2\" log\n+'\n+\n stop_git_daemon\n \n # Test protocol v2 with 'file://' transport\n@@ -80,4 +130,51 @@ test_expect_success 'ref advertisment is filtered with ls-remote using protocol\n \ttest_cmp actual expect\n '\n \n+test_expect_success 'clone with file:// using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -c protocol.version=2 \\\n+\t\tclone \"file://$(pwd)/file_parent\" file_child &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"clone< version 2\" log\n+'\n+\n+test_expect_success 'fetch with file:// using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_commit -C file_parent two &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -C file_child -c protocol.version=2 \\\n+\t\tfetch origin &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"fetch< version 2\" log\n+'\n+\n+test_expect_success 'ref advertisment is filtered during fetch using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_commit -C file_parent three &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -C file_child -c protocol.version=2 \\\n+\t\tfetch origin master &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t! grep \"refs/tags/one\" log &&\n+\t! grep \"refs/tags/two\" log &&\n+\t! grep \"refs/tags/three\" log\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex bf7ba6879..8e38352c5 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -256,14 +256,17 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tswitch (data->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\trefs = fetch_pack(&args, data->fd, data->conn,\n+\t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n+\t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n+\t\t\t\t &transport->pack_lockfile, data->version);\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n \t\trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n \t\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n-\t\t\t\t &transport->pack_lockfile);\n+\t\t\t\t &transport->pack_lockfile, data->version);\n \t\tbreak;\n \tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 2af6b1382..65a1beeb0 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -710,7 +710,6 @@ static void deepen(int depth, int deepen_relative,\n \t}\n \n \tsend_unshallow(shallows);\n-\tpacket_flush(1);\n }\n \n static void deepen_by_rev_list(int ac, const char **av,\n@@ -722,7 +721,53 @@ static void deepen_by_rev_list(int ac, const char **av,\n \tsend_shallow(result);\n \tfree_commit_list(result);\n \tsend_unshallow(shallows);\n-\tpacket_flush(1);\n+}\n+\n+/* Returns 1 if a shallow list is sent or 0 otherwise */\n+static int send_shallow_list(int depth, int deepen_rev_list,\n+\t\t\t timestamp_t deepen_since,\n+\t\t\t struct string_list *deepen_not,\n+\t\t\t struct object_array *shallows)\n+{\n+\tint ret = 0;\n+\n+\tif (depth > 0 && deepen_rev_list)\n+\t\tdie(\"git upload-pack: deepen and deepen-since (or deepen-not) cannot be used together\");\n+\tif (depth > 0) {\n+\t\tdeepen(depth, deepen_relative, shallows);\n+\t\tret = 1;\n+\t} else if (deepen_rev_list) {\n+\t\tstruct argv_array av = ARGV_ARRAY_INIT;\n+\t\tint i;\n+\n+\t\targv_array_push(&av, \"rev-list\");\n+\t\tif (deepen_since)\n+\t\t\targv_array_pushf(&av, \"--max-age=%\"PRItime, deepen_since);\n+\t\tif (deepen_not->nr) {\n+\t\t\targv_array_push(&av, \"--not\");\n+\t\t\tfor (i = 0; i < deepen_not->nr; i++) {\n+\t\t\t\tstruct string_list_item *s = deepen_not->items + i;\n+\t\t\t\targv_array_push(&av, s->string);\n+\t\t\t}\n+\t\t\targv_array_push(&av, \"--not\");\n+\t\t}\n+\t\tfor (i = 0; i < want_obj.nr; i++) {\n+\t\t\tstruct object *o = want_obj.objects[i].item;\n+\t\t\targv_array_push(&av, oid_to_hex(&o->oid));\n+\t\t}\n+\t\tdeepen_by_rev_list(av.argc, av.argv, shallows);\n+\t\targv_array_clear(&av);\n+\t\tret = 1;\n+\t} else {\n+\t\tif (shallows->nr > 0) {\n+\t\t\tint i;\n+\t\t\tfor (i = 0; i < shallows->nr; i++)\n+\t\t\t\tregister_shallow(&shallows->objects[i].item->oid);\n+\t\t}\n+\t}\n+\n+\tshallow_nr += shallows->nr;\n+\treturn ret;\n }\n \n static int process_shallow(const char *line, struct object_array *shallows)\n@@ -884,40 +929,10 @@ static void receive_needs(void)\n \n \tif (depth == 0 && !deepen_rev_list && shallows.nr == 0)\n \t\treturn;\n-\tif (depth > 0 && deepen_rev_list)\n-\t\tdie(\"git upload-pack: deepen and deepen-since (or deepen-not) cannot be used together\");\n-\tif (depth > 0)\n-\t\tdeepen(depth, deepen_relative, &shallows);\n-\telse if (deepen_rev_list) {\n-\t\tstruct argv_array av = ARGV_ARRAY_INIT;\n-\t\tint i;\n \n-\t\targv_array_push(&av, \"rev-list\");\n-\t\tif (deepen_since)\n-\t\t\targv_array_pushf(&av, \"--max-age=%\"PRItime, deepen_since);\n-\t\tif (deepen_not.nr) {\n-\t\t\targv_array_push(&av, \"--not\");\n-\t\t\tfor (i = 0; i < deepen_not.nr; i++) {\n-\t\t\t\tstruct string_list_item *s = deepen_not.items + i;\n-\t\t\t\targv_array_push(&av, s->string);\n-\t\t\t}\n-\t\t\targv_array_push(&av, \"--not\");\n-\t\t}\n-\t\tfor (i = 0; i < want_obj.nr; i++) {\n-\t\t\tstruct object *o = want_obj.objects[i].item;\n-\t\t\targv_array_push(&av, oid_to_hex(&o->oid));\n-\t\t}\n-\t\tdeepen_by_rev_list(av.argc, av.argv, &shallows);\n-\t\targv_array_clear(&av);\n-\t}\n-\telse\n-\t\tif (shallows.nr > 0) {\n-\t\t\tint i;\n-\t\t\tfor (i = 0; i < shallows.nr; i++)\n-\t\t\t\tregister_shallow(&shallows.objects[i].item->oid);\n-\t\t}\n-\n-\tshallow_nr += shallows.nr;\n+\tif (send_shallow_list(depth, deepen_rev_list, deepen_since,\n+\t\t\t &deepen_not, &shallows))\n+\t\tpacket_flush(1);\n \tobject_array_clear(&shallows);\n }\n \n@@ -1071,6 +1086,13 @@ struct upload_pack_data {\n \tstruct object_array wants;\n \tstruct oid_array haves;\n \n+\tstruct object_array shallows;\n+\tstruct string_list deepen_not;\n+\tint depth;\n+\ttimestamp_t deepen_since;\n+\tint deepen_rev_list;\n+\tint deepen_relative;\n+\n \tunsigned stateless_rpc : 1;\n \n \tunsigned use_thin_pack : 1;\n@@ -1084,16 +1106,22 @@ static void upload_pack_data_init(struct upload_pack_data *data)\n {\n \tstruct object_array wants = OBJECT_ARRAY_INIT;\n \tstruct oid_array haves = OID_ARRAY_INIT;\n+\tstruct object_array shallows = OBJECT_ARRAY_INIT;\n+\tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n \n \tmemset(data, 0, sizeof(*data));\n \tdata->wants = wants;\n \tdata->haves = haves;\n+\tdata->shallows = shallows;\n+\tdata->deepen_not = deepen_not;\n }\n \n static void upload_pack_data_clear(struct upload_pack_data *data)\n {\n \tobject_array_clear(&data->wants);\n \toid_array_clear(&data->haves);\n+\tobject_array_clear(&data->shallows);\n+\tstring_list_clear(&data->deepen_not, 0);\n }\n \n static int parse_want(const char *line)\n@@ -1178,6 +1206,22 @@ static void process_args(struct argv_array *args, struct upload_pack_data *data)\n \t\t\tcontinue;\n \t\t}\n \n+\t\t/* Shallow related arguments */\n+\t\tif (process_shallow(arg, &data->shallows))\n+\t\t\tcontinue;\n+\t\tif (process_deepen(arg, &data->depth))\n+\t\t\tcontinue;\n+\t\tif (process_deepen_since(arg, &data->deepen_since,\n+\t\t\t\t\t &data->deepen_rev_list))\n+\t\t\tcontinue;\n+\t\tif (process_deepen_not(arg, &data->deepen_not,\n+\t\t\t\t &data->deepen_rev_list))\n+\t\t\tcontinue;\n+\t\tif (!strcmp(arg, \"deepen-relative\")) {\n+\t\t\tdata->deepen_relative = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\t/* ignore unknown lines maybe? */\n \t\tdie(\"unexpect line: '%s'\", arg);\n \t}\n@@ -1273,6 +1317,23 @@ static int process_haves_and_send_acks(struct upload_pack_data *data)\n \treturn ret;\n }\n \n+static void send_shallow_info(struct upload_pack_data *data)\n+{\n+\t/* No shallow info needs to be sent */\n+\tif (!data->depth && !data->deepen_rev_list && !data->shallows.nr &&\n+\t !is_repository_shallow())\n+\t\treturn;\n+\n+\tpacket_write_fmt(1, \"shallow-info\\n\");\n+\n+\tif (!send_shallow_list(data->depth, data->deepen_rev_list,\n+\t\t\t data->deepen_since, &data->deepen_not,\n+\t\t\t &data->shallows) && is_repository_shallow())\n+\t\tdeepen(INFINITE_DEPTH, data->deepen_relative, &data->shallows);\n+\n+\tpacket_delim(1);\n+}\n+\n enum fetch_state {\n \tFETCH_PROCESS_ARGS = 0,\n \tFETCH_SEND_ACKS,\n@@ -1320,6 +1381,8 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys,\n \t\t\t\tstate = FETCH_DONE;\n \t\t\tbreak;\n \t\tcase FETCH_SEND_PACK:\n+\t\t\tsend_shallow_info(&data);\n+\n \t\t\tpacket_write_fmt(1, \"packfile\\n\");\n \t\t\tcreate_pack_file();\n \t\t\tstate = FETCH_DONE;\n@@ -1332,3 +1395,11 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys,\n \tupload_pack_data_clear(&data);\n \treturn 0;\n }\n+\n+int upload_pack_advertise(struct repository *r,\n+\t\t\t struct strbuf *value)\n+{\n+\tif (value)\n+\t\tstrbuf_addstr(value, \"shallow\");\n+\treturn 1;\n+}\ndiff --git a/upload-pack.h b/upload-pack.h\nindex 6b7890238..7720f2142 100644\n--- a/upload-pack.h\n+++ b/upload-pack.h\n@@ -14,5 +14,8 @@ struct repository;\n struct argv_array;\n extern int upload_pack_v2(struct repository *r, struct argv_array *keys,\n \t\t\t struct argv_array *args);\n+struct strbuf;\n+extern int upload_pack_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value);\n \n #endif /* UPLOAD_PACK_H */\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340660","messageId":"20180228232252.102167-23-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 22/35] fetch-pack: support shallow requests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:39Z","receivedAt":"2018-02-28T23:24:28Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Enable shallow clones and deepen requests using protocol version 2 if\nthe server 'fetch' command supports the 'shallow' feature.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 22 ++++++++++++++++\n connect.h | 2 ++\n fetch-pack.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n 3 files changed, 94 insertions(+), 1 deletion(-)\n\ndiff --git a/connect.c b/connect.c\nindex 6203ce576..66a9443c8 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -82,6 +82,28 @@ int server_supports_v2(const char *c, int die_on_error)\n \treturn 0;\n }\n \n+int server_supports_feature(const char *c, const char *feature,\n+\t\t\t int die_on_error)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < server_capabilities_v2.argc; i++) {\n+\t\tconst char *out;\n+\t\tif (skip_prefix(server_capabilities_v2.argv[i], c, &out) &&\n+\t\t (!*out || *(out++) == '=')) {\n+\t\t\tif (parse_feature_request(out, feature))\n+\t\t\t\treturn 1;\n+\t\t\telse\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (die_on_error)\n+\t\tdie(\"server doesn't support feature '%s'\", feature);\n+\n+\treturn 0;\n+}\n+\n static void process_capabilities_v2(struct packet_reader *reader)\n {\n \twhile (packet_reader_read(reader) == PACKET_READ_NORMAL)\ndiff --git a/connect.h b/connect.h\nindex 8898d4495..0e69c6709 100644\n--- a/connect.h\n+++ b/connect.h\n@@ -17,5 +17,7 @@ struct packet_reader;\n extern enum protocol_version discover_version(struct packet_reader *reader);\n \n extern int server_supports_v2(const char *c, int die_on_error);\n+extern int server_supports_feature(const char *c, const char *feature,\n+\t\t\t\t int die_on_error);\n \n #endif\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex dffcfd66a..837e1fd21 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1008,6 +1008,26 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \treturn ref;\n }\n \n+static void add_shallow_requests(struct strbuf *req_buf,\n+\t\t\t\t const struct fetch_pack_args *args)\n+{\n+\tif (is_repository_shallow())\n+\t\twrite_shallow_commits(req_buf, 1, NULL);\n+\tif (args->depth > 0)\n+\t\tpacket_buf_write(req_buf, \"deepen %d\", args->depth);\n+\tif (args->deepen_since) {\n+\t\ttimestamp_t max_age = approxidate(args->deepen_since);\n+\t\tpacket_buf_write(req_buf, \"deepen-since %\"PRItime, max_age);\n+\t}\n+\tif (args->deepen_not) {\n+\t\tint i;\n+\t\tfor (i = 0; i < args->deepen_not->nr; i++) {\n+\t\t\tstruct string_list_item *s = args->deepen_not->items + i;\n+\t\t\tpacket_buf_write(req_buf, \"deepen-not %s\", s->string);\n+\t\t}\n+\t}\n+}\n+\n static void add_wants(const struct ref *wants, struct strbuf *req_buf)\n {\n \tfor ( ; wants ; wants = wants->next) {\n@@ -1093,6 +1113,12 @@ static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n \tif (prefer_ofs_delta)\n \t\tpacket_buf_write(&req_buf, \"ofs-delta\");\n \n+\t/* Add shallow-info and deepen request */\n+\tif (server_supports_feature(\"fetch\", \"shallow\", 0))\n+\t\tadd_shallow_requests(&req_buf, args);\n+\telse if (is_repository_shallow() || args->deepen)\n+\t\tdie(_(\"Server does not support shallow requests\"));\n+\n \t/* add wants */\n \tadd_wants(wants, &req_buf);\n \n@@ -1122,7 +1148,7 @@ static int process_section_header(struct packet_reader *reader,\n \tint ret;\n \n \tif (packet_reader_peek(reader) != PACKET_READ_NORMAL)\n-\t\tdie(\"error reading packet\");\n+\t\tdie(\"error reading section header '%s'\", section);\n \n \tret = !strcmp(reader->line, section);\n \n@@ -1177,6 +1203,43 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n \treturn received_ready ? 2 : (received_ack ? 1 : 0);\n }\n \n+static void receive_shallow_info(struct fetch_pack_args *args,\n+\t\t\t\t struct packet_reader *reader)\n+{\n+\tprocess_section_header(reader, \"shallow-info\", 0);\n+\twhile (packet_reader_read(reader) == PACKET_READ_NORMAL) {\n+\t\tconst char *arg;\n+\t\tstruct object_id oid;\n+\n+\t\tif (skip_prefix(reader->line, \"shallow \", &arg)) {\n+\t\t\tif (get_oid_hex(arg, &oid))\n+\t\t\t\tdie(_(\"invalid shallow line: %s\"), reader->line);\n+\t\t\tregister_shallow(&oid);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(reader->line, \"unshallow \", &arg)) {\n+\t\t\tif (get_oid_hex(arg, &oid))\n+\t\t\t\tdie(_(\"invalid unshallow line: %s\"), reader->line);\n+\t\t\tif (!lookup_object(oid.hash))\n+\t\t\t\tdie(_(\"object not found: %s\"), reader->line);\n+\t\t\t/* make sure that it is parsed as shallow */\n+\t\t\tif (!parse_object(&oid))\n+\t\t\t\tdie(_(\"error in object: %s\"), reader->line);\n+\t\t\tif (unregister_shallow(&oid))\n+\t\t\t\tdie(_(\"no shallow found: %s\"), reader->line);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tdie(_(\"expected shallow/unshallow, got %s\"), reader->line);\n+\t}\n+\n+\tif (reader->status != PACKET_READ_FLUSH &&\n+\t reader->status != PACKET_READ_DELIM)\n+\t\tdie(\"error processing shallow info: %d\", reader->status);\n+\n+\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file, NULL);\n+\targs->deepen = 1;\n+}\n+\n enum fetch_state {\n \tFETCH_CHECK_LOCAL = 0,\n \tFETCH_SEND_REQUEST,\n@@ -1209,6 +1272,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t/* v2 supports these by default */\n \t\t\tallow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1;\n \t\t\tuse_sideband = 2;\n+\t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n+\t\t\t\targs->deepen = 1;\n \n \t\t\tif (marked)\n \t\t\t\tfor_each_ref(clear_marks, NULL);\n@@ -1245,6 +1310,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t}\n \t\t\tbreak;\n \t\tcase FETCH_GET_PACK:\n+\t\t\t/* Check for shallow-info section */\n+\t\t\tif (process_section_header(&reader, \"shallow-info\", 1))\n+\t\t\t\treceive_shallow_info(args, &reader);\n+\n \t\t\t/* get the pack */\n \t\t\tprocess_section_header(&reader, \"packfile\", 0);\n \t\t\tif (get_pack(args, fd, pack_lockfile))\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340661","messageId":"20180228232252.102167-24-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 23/35] connect: refactor git_connect to only get the protocol version once","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:40Z","receivedAt":"2018-02-28T23:24:30Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Instead of having each builtin transport asking for which protocol\nversion the user has configured in 'protocol.version' by calling\n`get_protocol_version_config()` multiple times, factor this logic out\nso there is just a single call at the beginning of `git_connect()`.\n\nThis will be helpful in the next patch where we can have centralized\nlogic which determines if we need to request a different protocol\nversion than what the user has configured.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 27 +++++++++++++++------------\n 1 file changed, 15 insertions(+), 12 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 66a9443c8..a0bfcdf4f 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -1035,6 +1035,7 @@ static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n */\n static struct child_process *git_connect_git(int fd[2], char *hostandport,\n \t\t\t\t\t const char *path, const char *prog,\n+\t\t\t\t\t enum protocol_version version,\n \t\t\t\t\t int flags)\n {\n \tstruct child_process *conn;\n@@ -1073,10 +1074,10 @@ static struct child_process *git_connect_git(int fd[2], char *hostandport,\n \t\t target_host, 0);\n \n \t/* If using a new version put that stuff here after a second null byte */\n-\tif (get_protocol_version_config() > 0) {\n+\tif (version > 0) {\n \t\tstrbuf_addch(&request, '\\0');\n \t\tstrbuf_addf(&request, \"version=%d%c\",\n-\t\t\t get_protocol_version_config(), '\\0');\n+\t\t\t version, '\\0');\n \t}\n \n \tpacket_write(fd[1], request.buf, request.len);\n@@ -1092,14 +1093,14 @@ static struct child_process *git_connect_git(int fd[2], char *hostandport,\n */\n static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n \t\t\t enum ssh_variant variant, const char *port,\n-\t\t\t int flags)\n+\t\t\t enum protocol_version version, int flags)\n {\n \tif (variant == VARIANT_SSH &&\n-\t get_protocol_version_config() > 0) {\n+\t version > 0) {\n \t\targv_array_push(args, \"-o\");\n \t\targv_array_push(args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n \t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n-\t\t\t\t get_protocol_version_config());\n+\t\t\t\t version);\n \t}\n \n \tif (flags & CONNECT_IPV4) {\n@@ -1152,7 +1153,8 @@ static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n \n /* Prepare a child_process for use by Git's SSH-tunneled transport. */\n static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n-\t\t\t const char *port, int flags)\n+\t\t\t const char *port, enum protocol_version version,\n+\t\t\t int flags)\n {\n \tconst char *ssh;\n \tenum ssh_variant variant;\n@@ -1186,14 +1188,14 @@ static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n \t\targv_array_push(&detect.args, ssh);\n \t\targv_array_push(&detect.args, \"-G\");\n \t\tpush_ssh_options(&detect.args, &detect.env_array,\n-\t\t\t\t VARIANT_SSH, port, flags);\n+\t\t\t\t VARIANT_SSH, port, version, flags);\n \t\targv_array_push(&detect.args, ssh_host);\n \n \t\tvariant = run_command(&detect) ? VARIANT_SIMPLE : VARIANT_SSH;\n \t}\n \n \targv_array_push(&conn->args, ssh);\n-\tpush_ssh_options(&conn->args, &conn->env_array, variant, port, flags);\n+\tpush_ssh_options(&conn->args, &conn->env_array, variant, port, version, flags);\n \targv_array_push(&conn->args, ssh_host);\n }\n \n@@ -1214,6 +1216,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \tchar *hostandport, *path;\n \tstruct child_process *conn;\n \tenum protocol protocol;\n+\tenum protocol_version version = get_protocol_version_config();\n \n \t/* Without this we cannot rely on waitpid() to tell\n \t * what happened to our children.\n@@ -1228,7 +1231,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tprintf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n \t\tconn = NULL;\n \t} else if (protocol == PROTO_GIT) {\n-\t\tconn = git_connect_git(fd, hostandport, path, prog, flags);\n+\t\tconn = git_connect_git(fd, hostandport, path, prog, version, flags);\n \t} else {\n \t\tstruct strbuf cmd = STRBUF_INIT;\n \t\tconst char *const *var;\n@@ -1271,12 +1274,12 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tstrbuf_release(&cmd);\n \t\t\t\treturn NULL;\n \t\t\t}\n-\t\t\tfill_ssh_args(conn, ssh_host, port, flags);\n+\t\t\tfill_ssh_args(conn, ssh_host, port, version, flags);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\n-\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\tif (version > 0) {\n \t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n-\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t\t\t\t version);\n \t\t\t}\n \t\t}\n \t\targv_array_push(&conn->args, cmd.buf);\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340662","messageId":"20180228232252.102167-25-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 24/35] connect: don't request v2 when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:41Z","receivedAt":"2018-02-28T23:24:35Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to be able to ship protocol v2 with only supporting fetch, we\nneed clients to not issue a request to use protocol v2 when pushing\n(since the client currently doesn't know how to push using protocol v2).\nThis allows a client to have protocol v2 configured in\n`protocol.version` and take advantage of using v2 for fetch and falling\nback to using v0 when pushing while v2 for push is being designed.\n\nWe could run into issues if we didn't fall back to protocol v2 when\npushing right now. This is because currently a server will ignore a request to\nuse v2 when contacting the 'receive-pack' endpoint and fall back to\nusing v0, but when push v2 is rolled out to servers, the 'receive-pack'\nendpoint will start responding using v2. So we don't want to get into a\nstate where a client is requesting to push with v2 before they actually\nknow how to push using v2.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 8 ++++++++\n t/t5702-protocol-v2.sh | 24 ++++++++++++++++++++++++\n 2 files changed, 32 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex a0bfcdf4f..32284d050 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -1218,6 +1218,14 @@ struct child_process *git_connect(int fd[2], const char *url,\n \tenum protocol protocol;\n \tenum protocol_version version = get_protocol_version_config();\n \n+\t/*\n+\t * NEEDSWORK: If we are trying to use protocol v2 and we are planning\n+\t * to perform a push, then fallback to v0 since the client doesn't know\n+\t * how to push yet using v2.\n+\t */\n+\tif (version == protocol_v2 && !strcmp(\"git-receive-pack\", prog))\n+\t\tversion = protocol_v0;\n+\n \t/* Without this we cannot rely on waitpid() to tell\n \t * what happened to our children.\n \t */\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 4365ac273..e3a7c09d4 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -95,6 +95,30 @@ test_expect_success 'pull with git:// using protocol v2' '\n \tgrep \"fetch< version 2\" log\n '\n \n+test_expect_success 'push with git:// and a config of v2 does not request v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\t# Till v2 for push is designed, make sure that if a client has\n+\t# protocol.version configured to use v2, that the client instead falls\n+\t# back and uses v0.\n+\n+\ttest_commit -C daemon_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -C daemon_child -c protocol.version=2 \\\n+\t\tpush origin HEAD:client_branch &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\t! grep \"push> .*\\\\\\0\\\\\\0version=2\\\\\\0$\" log &&\n+\t# Server responded using protocol v2\n+\t! grep \"push< version 2\" log\n+'\n+\n stop_git_daemon\n \n # Test protocol v2 with 'file://' transport\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340663","messageId":"20180228232252.102167-26-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 25/35] transport-helper: remove name parameter","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:42Z","receivedAt":"2018-02-28T23:24:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Commit 266f1fdfa (transport-helper: be quiet on read errors from\nhelpers, 2013-06-21) removed a call to 'die()' which printed the name of\nthe remote helper passed in to the 'recvline_fh()' function using the\n'name' parameter. Once the call to 'die()' was removed the parameter\nwas no longer necessary but wasn't removed. Clean up 'recvline_fh()'\nparameter list by removing the 'name' parameter.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 4c334b5ee..d72155768 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -49,7 +49,7 @@ static void sendline(struct helper_data *helper, struct strbuf *buffer)\n \t\tdie_errno(\"Full write to remote helper failed\");\n }\n \n-static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)\n+static int recvline_fh(FILE *helper, struct strbuf *buffer)\n {\n \tstrbuf_reset(buffer);\n \tif (debug)\n@@ -67,7 +67,7 @@ static int recvline_fh(FILE *helper, struct strbuf *buffer, const char *name)\n \n static int recvline(struct helper_data *helper, struct strbuf *buffer)\n {\n-\treturn recvline_fh(helper->out, buffer, helper->name);\n+\treturn recvline_fh(helper->out, buffer);\n }\n \n static void write_constant(int fd, const char *str)\n@@ -586,7 +586,7 @@ static int process_connect_service(struct transport *transport,\n \t\tgoto exit;\n \n \tsendline(data, &cmdbuf);\n-\tif (recvline_fh(input, &cmdbuf, name))\n+\tif (recvline_fh(input, &cmdbuf))\n \t\texit(128);\n \n \tif (!strcmp(cmdbuf.buf, \"\")) {\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340664","messageId":"20180228232252.102167-28-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 27/35] transport-helper: introduce stateless-connect","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:44Z","receivedAt":"2018-02-28T23:24:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Introduce the transport-helper capability 'stateless-connect'. This\ncapability indicates that the transport-helper can be requested to run\nthe 'stateless-connect' command which should attempt to make a\nstateless connection with a remote end. Once established, the\nconnection can be used by the git client to communicate with\nthe remote end natively in a stateless-rpc manner as supported by\nprotocol v2. This means that the client must send everything the server\nneeds in a single request as the client must not assume any\nstate-storing on the part of the server or transport.\n\nIf a stateless connection cannot be established then the remote-helper\nwill respond in the same manner as the 'connect' command indicating that\nthe client should fallback to using the dumb remote-helper commands.\n\nA future patch will implement the 'stateless-connect' capability in our\nhttp remote-helper (remote-curl) so that protocol v2 can be used using\nthe http transport.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/gitremote-helpers.txt | 32 +++++++++++++++++++++++++++++\n transport-helper.c | 11 ++++++++++\n transport.c | 1 +\n transport.h | 6 ++++++\n 4 files changed, 50 insertions(+)\n\ndiff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt\nindex 4a584f3c5..a8361ed95 100644\n--- a/Documentation/gitremote-helpers.txt\n+++ b/Documentation/gitremote-helpers.txt\n@@ -102,6 +102,14 @@ Capabilities for Pushing\n +\n Supported commands: 'connect'.\n \n+'stateless-connect'::\n+\tExperimental; for internal use only.\n+\tCan attempt to connect to a remote server for communication\n+\tusing git's wire-protocol version 2. This establishes a\n+\tstateless, half-duplex connection.\n++\n+Supported commands: 'stateless-connect'.\n+\n 'push'::\n \tCan discover remote refs and push local commits and the\n \thistory leading up to them to new or existing remote refs.\n@@ -136,6 +144,14 @@ Capabilities for Fetching\n +\n Supported commands: 'connect'.\n \n+'stateless-connect'::\n+\tExperimental; for internal use only.\n+\tCan attempt to connect to a remote server for communication\n+\tusing git's wire-protocol version 2. This establishes a\n+\tstateless, half-duplex connection.\n++\n+Supported commands: 'stateless-connect'.\n+\n 'fetch'::\n \tCan discover remote refs and transfer objects reachable from\n \tthem to the local object store.\n@@ -375,6 +391,22 @@ Supported if the helper has the \"export\" capability.\n +\n Supported if the helper has the \"connect\" capability.\n \n+'stateless-connect' ::\n+\tExperimental; for internal use only.\n+\tConnects to the given remote service for communication using\n+\tgit's wire-protocol version 2. This establishes a stateless,\n+\thalf-duplex connection. Valid replies to this command are empty\n+\tline (connection established), 'fallback' (no smart transport\n+\tsupport, fall back to dumb transports) and just exiting with\n+\terror message printed (can't connect, don't bother trying to\n+\tfall back). After line feed terminating the positive (empty)\n+\tresponse, the output of the service starts. Messages (both\n+\trequest and response) must be terminated with a single flush\n+\tpacket, allowing the remote helper to properly act as a proxy.\n+\tAfter the connection ends, the remote helper exits.\n++\n+Supported if the helper has the \"stateless-connect\" capability.\n+\n If a fatal error occurs, the program writes the error message to\n stderr and exits. The caller should expect that a suitable error\n message has been printed if the child closes the connection without\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c032a2a87..e20a5076e 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -12,6 +12,7 @@\n #include \"argv-array.h\"\n #include \"refs.h\"\n #include \"transport-internal.h\"\n+#include \"protocol.h\"\n \n static int debug;\n \n@@ -26,6 +27,7 @@ struct helper_data {\n \t\toption : 1,\n \t\tpush : 1,\n \t\tconnect : 1,\n+\t\tstateless_connect : 1,\n \t\tsigned_tags : 1,\n \t\tcheck_connectivity : 1,\n \t\tno_disconnect_req : 1,\n@@ -188,6 +190,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\trefspecs[refspec_nr++] = xstrdup(arg);\n \t\t} else if (!strcmp(capname, \"connect\")) {\n \t\t\tdata->connect = 1;\n+\t\t} else if (!strcmp(capname, \"stateless-connect\")) {\n+\t\t\tdata->stateless_connect = 1;\n \t\t} else if (!strcmp(capname, \"signed-tags\")) {\n \t\t\tdata->signed_tags = 1;\n \t\t} else if (skip_prefix(capname, \"export-marks \", &arg)) {\n@@ -612,6 +616,13 @@ static int process_connect_service(struct transport *transport,\n \tif (data->connect) {\n \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n \t\tret = run_connect(transport, &cmdbuf);\n+\t} else if (data->stateless_connect &&\n+\t\t (get_protocol_version_config() == protocol_v2) &&\n+\t\t !strcmp(\"git-upload-pack\", name)) {\n+\t\tstrbuf_addf(&cmdbuf, \"stateless-connect %s\\n\", name);\n+\t\tret = run_connect(transport, &cmdbuf);\n+\t\tif (ret)\n+\t\t\ttransport->stateless_rpc = 1;\n \t}\n \n \tstrbuf_release(&cmdbuf);\ndiff --git a/transport.c b/transport.c\nindex 8e38352c5..2e7b7a715 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -250,6 +250,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n+\targs.stateless_rpc = transport->stateless_rpc;\n \n \tif (!data->got_remote_heads)\n \t\trefs_tmp = get_refs_via_connect(transport, 0, NULL);\ndiff --git a/transport.h b/transport.h\nindex daea4770c..0ef0d1902 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -55,6 +55,12 @@ struct transport {\n \t */\n \tunsigned cloning : 1;\n \n+\t/*\n+\t * Indicates that the transport is connected via a half-duplex\n+\t * connection and should operate in stateless-rpc mode.\n+\t */\n+\tunsigned stateless_rpc : 1;\n+\n \t/*\n \t * These strings will be passed to the {pre, post}-receive hook,\n \t * on the remote side, if both sides support the push options capability.\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340665","messageId":"20180228232252.102167-30-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 29/35] remote-curl: create copy of the service name","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:46Z","receivedAt":"2018-02-28T23:24:43Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Make a copy of the service name being requested instead of relying on\nthe buffer pointed to by the passed in 'const char *' to remain\nunchanged.\n\nCurrently, all service names are string constants, but a subsequent\npatch will introduce service names from external sources.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex dae8a4a48..4086aa733 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -165,7 +165,7 @@ static int set_option(const char *name, const char *value)\n }\n \n struct discovery {\n-\tconst char *service;\n+\tchar *service;\n \tchar *buf_alloc;\n \tchar *buf;\n \tsize_t len;\n@@ -257,6 +257,7 @@ static void free_discovery(struct discovery *d)\n \t\tfree(d->shallow.oid);\n \t\tfree(d->buf_alloc);\n \t\tfree_refs(d->refs);\n+\t\tfree(d->service);\n \t\tfree(d);\n \t}\n }\n@@ -343,7 +344,7 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \t\twarning(_(\"redirecting to %s\"), url.buf);\n \n \tlast= xcalloc(1, sizeof(*last_discovery));\n-\tlast->service = service;\n+\tlast->service = xstrdup(service);\n \tlast->buf_alloc = strbuf_detach(&buffer, &last->len);\n \tlast->buf = last->buf_alloc;\n \n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340666","messageId":"20180228232252.102167-29-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 28/35] pkt-line: add packet_buf_write_len function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:45Z","receivedAt":"2018-02-28T23:24:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add the 'packet_buf_write_len()' function which allows for writing an\narbitrary length buffer into a 'struct strbuf' and formatting it in\npacket-line format.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 16 ++++++++++++++++\n pkt-line.h | 1 +\n 2 files changed, 17 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 87a24bd17..5223e24e2 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -215,6 +215,22 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n \tva_end(args);\n }\n \n+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len)\n+{\n+\tsize_t orig_len, n;\n+\n+\torig_len = buf->len;\n+\tstrbuf_addstr(buf, \"0000\");\n+\tstrbuf_add(buf, data, len);\n+\tn = buf->len - orig_len;\n+\n+\tif (n > LARGE_PACKET_MAX)\n+\t\tdie(\"protocol error: impossibly long line\");\n+\n+\tset_packet_header(&buf->buf[orig_len], n);\n+\tpacket_trace(data, len, 1);\n+}\n+\n int write_packetized_from_fd(int fd_in, int fd_out)\n {\n \tstatic char buf[LARGE_PACKET_DATA_MAX];\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 3f836f01a..4f97ae3e5 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -26,6 +26,7 @@ void packet_buf_flush(struct strbuf *buf);\n void packet_buf_delim(struct strbuf *buf);\n void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+void packet_buf_write_len(struct strbuf *buf, const char *data, size_t len);\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int write_packetized_from_fd(int fd_in, int fd_out);\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340667","messageId":"20180228232252.102167-32-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 31/35] http: allow providing extra headers for http requests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:48Z","receivedAt":"2018-02-28T23:24:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a way for callers to request that extra headers be included when\nmaking http requests.\n\nSigned-off-by: Brandon Williams \n---\n http.c | 8 ++++++++\n http.h | 7 +++++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/http.c b/http.c\nindex 597771271..e1757d62b 100644\n--- a/http.c\n+++ b/http.c\n@@ -1723,6 +1723,14 @@ static int http_request(const char *url,\n \n \theaders = curl_slist_append(headers, buf.buf);\n \n+\t/* Add additional headers here */\n+\tif (options && options->extra_headers) {\n+\t\tconst struct string_list_item *item;\n+\t\tfor_each_string_list_item(item, options->extra_headers) {\n+\t\t\theaders = curl_slist_append(headers, item->string);\n+\t\t}\n+\t}\n+\n \tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, \"gzip\");\ndiff --git a/http.h b/http.h\nindex f7bd3b26b..4df4a25e1 100644\n--- a/http.h\n+++ b/http.h\n@@ -172,6 +172,13 @@ struct http_get_options {\n \t * for details.\n \t */\n \tstruct strbuf *base_url;\n+\n+\t/*\n+\t * If not NULL, contains additional HTTP headers to be sent with the\n+\t * request. The strings in the list must not be freed until after the\n+\t * request has completed.\n+\t */\n+\tstruct string_list *extra_headers;\n };\n \n /* Return values for http_get_*() */\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340668","messageId":"20180228232252.102167-33-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 32/35] http: don't always add Git-Protocol header","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:49Z","receivedAt":"2018-02-28T23:24:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Instead of always sending the Git-Protocol header with the configured\nversion with every http request, explicitly send it when discovering\nrefs and then only send it on subsequent http requests if the server\nunderstood the version requested.\n\nSigned-off-by: Brandon Williams \n---\n http.c | 17 -----------------\n remote-curl.c | 33 +++++++++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 17 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex e1757d62b..8f1129ac7 100644\n--- a/http.c\n+++ b/http.c\n@@ -904,21 +904,6 @@ static void set_from_env(const char **var, const char *envname)\n \t\t*var = val;\n }\n \n-static void protocol_http_header(void)\n-{\n-\tif (get_protocol_version_config() > 0) {\n-\t\tstruct strbuf protocol_header = STRBUF_INIT;\n-\n-\t\tstrbuf_addf(&protocol_header, GIT_PROTOCOL_HEADER \": version=%d\",\n-\t\t\t get_protocol_version_config());\n-\n-\n-\t\textra_http_headers = curl_slist_append(extra_http_headers,\n-\t\t\t\t\t\t protocol_header.buf);\n-\t\tstrbuf_release(&protocol_header);\n-\t}\n-}\n-\n void http_init(struct remote *remote, const char *url, int proactive_auth)\n {\n \tchar *low_speed_limit;\n@@ -949,8 +934,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tif (remote)\n \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n \n-\tprotocol_http_header();\n-\n \tpragma_header = curl_slist_append(http_copy_default_headers(),\n \t\t\"Pragma: no-cache\");\n \tno_pragma_header = curl_slist_append(http_copy_default_headers(),\ndiff --git a/remote-curl.c b/remote-curl.c\nindex c54035843..b4e9db85b 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -291,6 +291,19 @@ static int show_http_message(struct strbuf *type, struct strbuf *charset,\n \treturn 0;\n }\n \n+static int get_protocol_http_header(enum protocol_version version,\n+\t\t\t\t struct strbuf *header)\n+{\n+\tif (version > 0) {\n+\t\tstrbuf_addf(header, GIT_PROTOCOL_HEADER \": version=%d\",\n+\t\t\t version);\n+\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static struct discovery *discover_refs(const char *service, int for_push)\n {\n \tstruct strbuf exp = STRBUF_INIT;\n@@ -299,6 +312,8 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \tstruct strbuf buffer = STRBUF_INIT;\n \tstruct strbuf refs_url = STRBUF_INIT;\n \tstruct strbuf effective_url = STRBUF_INIT;\n+\tstruct strbuf protocol_header = STRBUF_INIT;\n+\tstruct string_list extra_headers = STRING_LIST_INIT_DUP;\n \tstruct discovery *last = last_discovery;\n \tint http_ret, maybe_smart = 0;\n \tstruct http_get_options http_options;\n@@ -318,11 +333,16 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \t\tstrbuf_addf(&refs_url, \"service=%s\", service);\n \t}\n \n+\t/* Add the extra Git-Protocol header */\n+\tif (get_protocol_http_header(get_protocol_version_config(), &protocol_header))\n+\t\tstring_list_append(&extra_headers, protocol_header.buf);\n+\n \tmemset(&http_options, 0, sizeof(http_options));\n \thttp_options.content_type = &type;\n \thttp_options.charset = &charset;\n \thttp_options.effective_url = &effective_url;\n \thttp_options.base_url = &url;\n+\thttp_options.extra_headers = &extra_headers;\n \thttp_options.initial_request = 1;\n \thttp_options.no_cache = 1;\n \thttp_options.keep_error = 1;\n@@ -389,6 +409,8 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \tstrbuf_release(&charset);\n \tstrbuf_release(&effective_url);\n \tstrbuf_release(&buffer);\n+\tstrbuf_release(&protocol_header);\n+\tstring_list_clear(&extra_headers, 0);\n \tlast_discovery = last;\n \treturn last;\n }\n@@ -425,6 +447,7 @@ struct rpc_state {\n \tchar *service_url;\n \tchar *hdr_content_type;\n \tchar *hdr_accept;\n+\tchar *protocol_header;\n \tchar *buf;\n \tsize_t alloc;\n \tsize_t len;\n@@ -611,6 +634,10 @@ static int post_rpc(struct rpc_state *rpc)\n \theaders = curl_slist_append(headers, needs_100_continue ?\n \t\t\"Expect: 100-continue\" : \"Expect:\");\n \n+\t/* Add the extra Git-Protocol header */\n+\tif (rpc->protocol_header)\n+\t\theaders = curl_slist_append(headers, rpc->protocol_header);\n+\n retry:\n \tslot = get_active_slot();\n \n@@ -751,6 +778,11 @@ static int rpc_service(struct rpc_state *rpc, struct discovery *heads)\n \tstrbuf_addf(&buf, \"Accept: application/x-%s-result\", svc);\n \trpc->hdr_accept = strbuf_detach(&buf, NULL);\n \n+\tif (get_protocol_http_header(heads->version, &buf))\n+\t\trpc->protocol_header = strbuf_detach(&buf, NULL);\n+\telse\n+\t\trpc->protocol_header = NULL;\n+\n \twhile (!err) {\n \t\tint n = packet_read(rpc->out, NULL, NULL, rpc->buf, rpc->alloc, 0);\n \t\tif (!n)\n@@ -778,6 +810,7 @@ static int rpc_service(struct rpc_state *rpc, struct discovery *heads)\n \tfree(rpc->service_url);\n \tfree(rpc->hdr_content_type);\n \tfree(rpc->hdr_accept);\n+\tfree(rpc->protocol_header);\n \tfree(rpc->buf);\n \tstrbuf_release(&buf);\n \treturn err;\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340669","messageId":"20180228232252.102167-34-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 33/35] http: eliminate \"# service\" line when using protocol v2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:50Z","receivedAt":"2018-02-28T23:24:56Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"When an http info/refs request is made, requesting that protocol v2 be\nused, don't send a \"# service\" line since this line is not part of the\nv2 spec.\n\nSigned-off-by: Brandon Williams \n---\n http-backend.c | 8 ++++++--\n remote-curl.c | 3 +++\n 2 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex f3dc218b2..5d241e910 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -10,6 +10,7 @@\n #include \"url.h\"\n #include \"argv-array.h\"\n #include \"packfile.h\"\n+#include \"protocol.h\"\n \n static const char content_type[] = \"Content-Type\";\n static const char content_length[] = \"Content-Length\";\n@@ -466,8 +467,11 @@ static void get_info_refs(struct strbuf *hdr, char *arg)\n \t\thdr_str(hdr, content_type, buf.buf);\n \t\tend_headers(hdr);\n \n-\t\tpacket_write_fmt(1, \"# service=git-%s\\n\", svc->name);\n-\t\tpacket_flush(1);\n+\n+\t\tif (determine_protocol_version_server() != protocol_v2) {\n+\t\t\tpacket_write_fmt(1, \"# service=git-%s\\n\", svc->name);\n+\t\t\tpacket_flush(1);\n+\t\t}\n \n \t\targv[0] = svc->name;\n \t\trun_service(argv, 0);\ndiff --git a/remote-curl.c b/remote-curl.c\nindex b4e9db85b..66a53f74b 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -396,6 +396,9 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \t\t\t;\n \n \t\tlast->proto_git = 1;\n+\t} else if (maybe_smart &&\n+\t\t last->len > 5 && starts_with(last->buf + 4, \"version 2\")) {\n+\t\tlast->proto_git = 1;\n \t}\n \n \tif (last->proto_git)\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340670","messageId":"20180228232252.102167-36-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 35/35] remote-curl: don't request v2 when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:52Z","receivedAt":"2018-02-28T23:24:57Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"In order to be able to ship protocol v2 with only supporting fetch, we\nneed clients to not issue a request to use protocol v2 when pushing\n(since the client currently doesn't know how to push using protocol v2).\nThis allows a client to have protocol v2 configured in\n`protocol.version` and take advantage of using v2 for fetch and falling\nback to using v0 when pushing while v2 for push is being designed.\n\nWe could run into issues if we didn't fall back to protocol v2 when\npushing right now. This is because currently a server will ignore a request to\nuse v2 when contacting the 'receive-pack' endpoint and fall back to\nusing v0, but when push v2 is rolled out to servers, the 'receive-pack'\nendpoint will start responding using v2. So we don't want to get into a\nstate where a client is requesting to push with v2 before they actually\nknow how to push using v2.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 11 ++++++++++-\n t/t5702-protocol-v2.sh | 24 ++++++++++++++++++++++++\n 2 files changed, 34 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 3f882d766..379ab9b21 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -322,6 +322,7 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \tstruct discovery *last = last_discovery;\n \tint http_ret, maybe_smart = 0;\n \tstruct http_get_options http_options;\n+\tenum protocol_version version = get_protocol_version_config();\n \n \tif (last && !strcmp(service, last->service))\n \t\treturn last;\n@@ -338,8 +339,16 @@ static struct discovery *discover_refs(const char *service, int for_push)\n \t\tstrbuf_addf(&refs_url, \"service=%s\", service);\n \t}\n \n+\t/*\n+\t * NEEDSWORK: If we are trying to use protocol v2 and we are planning\n+\t * to perform a push, then fallback to v0 since the client doesn't know\n+\t * how to push yet using v2.\n+\t */\n+\tif (version == protocol_v2 && !strcmp(\"git-receive-pack\", service))\n+\t\tversion = protocol_v0;\n+\n \t/* Add the extra Git-Protocol header */\n-\tif (get_protocol_http_header(get_protocol_version_config(), &protocol_header))\n+\tif (get_protocol_http_header(version, &protocol_header))\n \t\tstring_list_append(&extra_headers, protocol_header.buf);\n \n \tmemset(&http_options, 0, sizeof(http_options));\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 124063c2c..56f7c3c32 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -244,6 +244,30 @@ test_expect_success 'fetch with http:// using protocol v2' '\n \tgrep \"git< version 2\" log\n '\n \n+test_expect_success 'push with http:// and a config of v2 does not request v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\t# Till v2 for push is designed, make sure that if a client has\n+\t# protocol.version configured to use v2, that the client instead falls\n+\t# back and uses v0.\n+\n+\ttest_commit -C http_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -C http_child -c protocol.version=2 \\\n+\t\tpush origin HEAD:client_branch &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client didnt request to use protocol v2\n+\t! grep \"Git-Protocol: version=2\" log &&\n+\t# Server didnt respond using protocol v2\n+\t! grep \"git< version 2\" log\n+'\n+\n+\n stop_httpd\n \n test_done\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340671","messageId":"20180228232252.102167-31-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 30/35] remote-curl: store the protocol version the server responded with","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:47Z","receivedAt":"2018-02-28T23:25:02Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Store the protocol version the server responded with when performing\ndiscovery. This will be used in a future patch to either change the\n'Git-Protocol' header sent in subsequent requests or to determine if a\nclient needs to fallback to using a different protocol version.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 4086aa733..c54035843 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -171,6 +171,7 @@ struct discovery {\n \tsize_t len;\n \tstruct ref *refs;\n \tstruct oid_array shallow;\n+\tenum protocol_version version;\n \tunsigned proto_git : 1;\n };\n static struct discovery *last_discovery;\n@@ -184,7 +185,8 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \t\t\t PACKET_READ_CHOMP_NEWLINE |\n \t\t\t PACKET_READ_GENTLE_ON_EOF);\n \n-\tswitch (discover_version(&reader)) {\n+\theads->version = discover_version(&reader);\n+\tswitch (heads->version) {\n \tcase protocol_v2:\n \t\tdie(\"support for protocol v2 not implemented yet\");\n \t\tbreak;\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340672","messageId":"20180228232252.102167-35-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 34/35] remote-curl: implement stateless-connect command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:51Z","receivedAt":"2018-02-28T23:25:04Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach remote-curl the 'stateless-connect' command which is used to\nestablish a stateless connection with servers which support protocol\nversion 2. This allows remote-curl to act as a proxy, allowing the git\nclient to communicate natively with a remote end, simply using\nremote-curl as a pass through to convert requests to http.\n\nSigned-off-by: Brandon Williams \n---\n remote-curl.c | 205 ++++++++++++++++++++++++++++++++++++++++-\n t/t5702-protocol-v2.sh | 45 +++++++++\n 2 files changed, 249 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 66a53f74b..3f882d766 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -188,7 +188,12 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n \theads->version = discover_version(&reader);\n \tswitch (heads->version) {\n \tcase protocol_v2:\n-\t\tdie(\"support for protocol v2 not implemented yet\");\n+\t\t/*\n+\t\t * Do nothing. This isn't a list of refs but rather a\n+\t\t * capability advertisement. Client would have run\n+\t\t * 'stateless-connect' so we'll dump this capability listing\n+\t\t * and let them request the refs themselves.\n+\t\t */\n \t\tbreak;\n \tcase protocol_v1:\n \tcase protocol_v0:\n@@ -1085,6 +1090,200 @@ static void parse_push(struct strbuf *buf)\n \tfree(specs);\n }\n \n+/*\n+ * Used to represent the state of a connection to an HTTP server when\n+ * communicating using git's wire-protocol version 2.\n+ */\n+struct proxy_state {\n+\tchar *service_name;\n+\tchar *service_url;\n+\tstruct curl_slist *headers;\n+\tstruct strbuf request_buffer;\n+\tint in;\n+\tint out;\n+\tstruct packet_reader reader;\n+\tsize_t pos;\n+\tint seen_flush;\n+};\n+\n+static void proxy_state_init(struct proxy_state *p, const char *service_name,\n+\t\t\t enum protocol_version version)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tmemset(p, 0, sizeof(*p));\n+\tp->service_name = xstrdup(service_name);\n+\n+\tp->in = 0;\n+\tp->out = 1;\n+\tstrbuf_init(&p->request_buffer, 0);\n+\n+\tstrbuf_addf(&buf, \"%s%s\", url.buf, p->service_name);\n+\tp->service_url = strbuf_detach(&buf, NULL);\n+\n+\tp->headers = http_copy_default_headers();\n+\n+\tstrbuf_addf(&buf, \"Content-Type: application/x-%s-request\", p->service_name);\n+\tp->headers = curl_slist_append(p->headers, buf.buf);\n+\tstrbuf_reset(&buf);\n+\n+\tstrbuf_addf(&buf, \"Accept: application/x-%s-result\", p->service_name);\n+\tp->headers = curl_slist_append(p->headers, buf.buf);\n+\tstrbuf_reset(&buf);\n+\n+\tp->headers = curl_slist_append(p->headers, \"Transfer-Encoding: chunked\");\n+\n+\t/* Add the Git-Protocol header */\n+\tif (get_protocol_http_header(version, &buf))\n+\t\tp->headers = curl_slist_append(p->headers, buf.buf);\n+\n+\tpacket_reader_init(&p->reader, p->in, NULL, 0,\n+\t\t\t PACKET_READ_GENTLE_ON_EOF);\n+\n+\tstrbuf_release(&buf);\n+}\n+\n+static void proxy_state_clear(struct proxy_state *p)\n+{\n+\tfree(p->service_name);\n+\tfree(p->service_url);\n+\tcurl_slist_free_all(p->headers);\n+\tstrbuf_release(&p->request_buffer);\n+}\n+\n+/*\n+ * CURLOPT_READFUNCTION callback function.\n+ * Attempts to copy over a single packet-line at a time into the\n+ * curl provided buffer.\n+ */\n+static size_t proxy_in(char *buffer, size_t eltsize,\n+\t\t size_t nmemb, void *userdata)\n+{\n+\tsize_t max;\n+\tstruct proxy_state *p = userdata;\n+\tsize_t avail = p->request_buffer.len - p->pos;\n+\n+\n+\tif (eltsize != 1)\n+\t\tBUG(\"curl read callback called with size = %zu != 1\", eltsize);\n+\tmax = nmemb;\n+\n+\tif (!avail) {\n+\t\tif (p->seen_flush) {\n+\t\t\tp->seen_flush = 0;\n+\t\t\treturn 0;\n+\t\t}\n+\n+\t\tstrbuf_reset(&p->request_buffer);\n+\t\tswitch (packet_reader_read(&p->reader)) {\n+\t\tcase PACKET_READ_EOF:\n+\t\t\tdie(\"unexpected EOF when reading from parent process\");\n+\t\tcase PACKET_READ_NORMAL:\n+\t\t\tpacket_buf_write_len(&p->request_buffer, p->reader.line,\n+\t\t\t\t\t p->reader.pktlen);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_DELIM:\n+\t\t\tpacket_buf_delim(&p->request_buffer);\n+\t\t\tbreak;\n+\t\tcase PACKET_READ_FLUSH:\n+\t\t\tpacket_buf_flush(&p->request_buffer);\n+\t\t\tp->seen_flush = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\tp->pos = 0;\n+\t\tavail = p->request_buffer.len;\n+\t}\n+\n+\tif (max < avail)\n+\t\tavail = max;\n+\tmemcpy(buffer, p->request_buffer.buf + p->pos, avail);\n+\tp->pos += avail;\n+\treturn avail;\n+}\n+\n+static size_t proxy_out(char *buffer, size_t eltsize,\n+\t\t\tsize_t nmemb, void *userdata)\n+{\n+\tsize_t size;\n+\tstruct proxy_state *p = userdata;\n+\n+\tif (eltsize != 1)\n+\t\tBUG(\"curl read callback called with size = %zu != 1\", eltsize);\n+\tsize = nmemb;\n+\n+\twrite_or_die(p->out, buffer, size);\n+\treturn size;\n+}\n+\n+/* Issues a request to the HTTP server configured in `p` */\n+static int proxy_request(struct proxy_state *p)\n+{\n+\tstruct active_request_slot *slot;\n+\n+\tslot = get_active_slot();\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, p->service_url);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, p->headers);\n+\n+\t/* Setup function to read request from client */\n+\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, proxy_in);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_READDATA, p);\n+\n+\t/* Setup function to write server response to client */\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, proxy_out);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, p);\n+\n+\tif (run_slot(slot, NULL) != HTTP_OK)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static int stateless_connect(const char *service_name)\n+{\n+\tstruct discovery *discover;\n+\tstruct proxy_state p;\n+\n+\t/*\n+\t * Run the info/refs request and see if the server supports protocol\n+\t * v2. If and only if the server supports v2 can we successfully\n+\t * establish a stateless connection, otherwise we need to tell the\n+\t * client to fallback to using other transport helper functions to\n+\t * complete their request.\n+\t */\n+\tdiscover = discover_refs(service_name, 0);\n+\tif (discover->version != protocol_v2) {\n+\t\tprintf(\"fallback\\n\");\n+\t\tfflush(stdout);\n+\t\treturn -1;\n+\t} else {\n+\t\t/* Stateless Connection established */\n+\t\tprintf(\"\\n\");\n+\t\tfflush(stdout);\n+\t}\n+\n+\tproxy_state_init(&p, service_name, discover->version);\n+\n+\t/*\n+\t * Dump the capability listing that we got from the server earlier\n+\t * during the info/refs request.\n+\t */\n+\twrite_or_die(p.out, discover->buf, discover->len);\n+\n+\t/* Peek the next packet line. Until we see EOF keep sending POSTs */\n+\twhile (packet_reader_peek(&p.reader) != PACKET_READ_EOF) {\n+\t\tif (proxy_request(&p)) {\n+\t\t\t/* We would have an err here */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tproxy_state_clear(&p);\n+\treturn 0;\n+}\n+\n int cmd_main(int argc, const char **argv)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -1153,12 +1352,16 @@ int cmd_main(int argc, const char **argv)\n \t\t\tfflush(stdout);\n \n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n+\t\t\tprintf(\"stateless-connect\\n\");\n \t\t\tprintf(\"fetch\\n\");\n \t\t\tprintf(\"option\\n\");\n \t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"check-connectivity\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n+\t\t} else if (skip_prefix(buf.buf, \"stateless-connect \", &arg)) {\n+\t\t\tif (!stateless_connect(arg))\n+\t\t\t\tbreak;\n \t\t} else {\n \t\t\terror(\"remote-curl: unknown command '%s' from git\", buf.buf);\n \t\t\treturn 1;\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex e3a7c09d4..124063c2c 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -201,4 +201,49 @@ test_expect_success 'ref advertisment is filtered during fetch using protocol v2\n \t! grep \"refs/tags/three\" log\n '\n \n+# Test protocol v2 with 'http://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one\n+'\n+\n+test_expect_success 'clone with http:// using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" GIT_TRACE_CURL=\"$(pwd)/log\" git -c protocol.version=2 \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v2\n+\tgrep \"Git-Protocol: version=2\" log &&\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log\n+'\n+\n+test_expect_success 'fetch with http:// using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -C http_child -c protocol.version=2 \\\n+\t\tfetch &&\n+\n+\tgit -C http_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v2\n+\tgrep \"git< version 2\" log\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340673","messageId":"20180228232252.102167-27-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 26/35] transport-helper: refactor process_connect_service","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:43Z","receivedAt":"2018-02-28T23:25:11Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"A future patch will need to take advantage of the logic which runs and\nprocesses the response of the connect command on a remote helper so\nfactor out this logic from 'process_connect_service()' and place it into\na helper function 'run_connect()'.\n\nSigned-off-by: Brandon Williams \n---\n transport-helper.c | 67 ++++++++++++++++++++++++++--------------------\n 1 file changed, 38 insertions(+), 29 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex d72155768..c032a2a87 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -545,14 +545,13 @@ static int fetch_with_import(struct transport *transport,\n \treturn 0;\n }\n \n-static int process_connect_service(struct transport *transport,\n-\t\t\t\t const char *name, const char *exec)\n+static int run_connect(struct transport *transport, struct strbuf *cmdbuf)\n {\n \tstruct helper_data *data = transport->data;\n-\tstruct strbuf cmdbuf = STRBUF_INIT;\n-\tstruct child_process *helper;\n-\tint r, duped, ret = 0;\n+\tint ret = 0;\n+\tint duped;\n \tFILE *input;\n+\tstruct child_process *helper;\n \n \thelper = get_helper(transport);\n \n@@ -568,44 +567,54 @@ static int process_connect_service(struct transport *transport,\n \tinput = xfdopen(duped, \"r\");\n \tsetvbuf(input, NULL, _IONBF, 0);\n \n+\tsendline(data, cmdbuf);\n+\tif (recvline_fh(input, cmdbuf))\n+\t\texit(128);\n+\n+\tif (!strcmp(cmdbuf->buf, \"\")) {\n+\t\tdata->no_disconnect_req = 1;\n+\t\tif (debug)\n+\t\t\tfprintf(stderr, \"Debug: Smart transport connection \"\n+\t\t\t\t\"ready.\\n\");\n+\t\tret = 1;\n+\t} else if (!strcmp(cmdbuf->buf, \"fallback\")) {\n+\t\tif (debug)\n+\t\t\tfprintf(stderr, \"Debug: Falling back to dumb \"\n+\t\t\t\t\"transport.\\n\");\n+\t} else {\n+\t\tdie(\"Unknown response to connect: %s\",\n+\t\t\tcmdbuf->buf);\n+\t}\n+\n+\tfclose(input);\n+\treturn ret;\n+}\n+\n+static int process_connect_service(struct transport *transport,\n+\t\t\t\t const char *name, const char *exec)\n+{\n+\tstruct helper_data *data = transport->data;\n+\tstruct strbuf cmdbuf = STRBUF_INIT;\n+\tint ret = 0;\n+\n \t/*\n \t * Handle --upload-pack and friends. This is fire and forget...\n \t * just warn if it fails.\n \t */\n \tif (strcmp(name, exec)) {\n-\t\tr = set_helper_option(transport, \"servpath\", exec);\n+\t\tint r = set_helper_option(transport, \"servpath\", exec);\n \t\tif (r > 0)\n \t\t\twarning(\"Setting remote service path not supported by protocol.\");\n \t\telse if (r < 0)\n \t\t\twarning(\"Invalid remote service path.\");\n \t}\n \n-\tif (data->connect)\n+\tif (data->connect) {\n \t\tstrbuf_addf(&cmdbuf, \"connect %s\\n\", name);\n-\telse\n-\t\tgoto exit;\n-\n-\tsendline(data, &cmdbuf);\n-\tif (recvline_fh(input, &cmdbuf))\n-\t\texit(128);\n-\n-\tif (!strcmp(cmdbuf.buf, \"\")) {\n-\t\tdata->no_disconnect_req = 1;\n-\t\tif (debug)\n-\t\t\tfprintf(stderr, \"Debug: Smart transport connection \"\n-\t\t\t\t\"ready.\\n\");\n-\t\tret = 1;\n-\t} else if (!strcmp(cmdbuf.buf, \"fallback\")) {\n-\t\tif (debug)\n-\t\t\tfprintf(stderr, \"Debug: Falling back to dumb \"\n-\t\t\t\t\"transport.\\n\");\n-\t} else\n-\t\tdie(\"Unknown response to connect: %s\",\n-\t\t\tcmdbuf.buf);\n+\t\tret = run_connect(transport, &cmdbuf);\n+\t}\n \n-exit:\n \tstrbuf_release(&cmdbuf);\n-\tfclose(input);\n \treturn ret;\n }\n \n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340674","messageId":"20180228232252.102167-17-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 16/35] transport: convert transport_get_remote_refs to take a list of ref patterns","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:33Z","receivedAt":"2018-02-28T23:25:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Convert 'transport_get_remote_refs()' to optionally take a list of ref\npatterns.\n\nSigned-off-by: Brandon Williams \n---\n builtin/clone.c | 2 +-\n builtin/fetch.c | 4 ++--\n builtin/ls-remote.c | 2 +-\n builtin/remote.c | 2 +-\n transport.c | 7 +++++--\n transport.h | 12 +++++++++++-\n 6 files changed, 21 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 284651797..6e77d993f 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1121,7 +1121,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (transport->smart_options && !deepen)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n-\trefs = transport_get_remote_refs(transport);\n+\trefs = transport_get_remote_refs(transport, NULL);\n \n \tif (refs) {\n \t\tmapped_refs = wanted_peer_refs(refs, refspec);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7bbcd26fa..850382f55 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -250,7 +250,7 @@ static void find_non_local_tags(struct transport *transport,\n \tstruct string_list_item *item = NULL;\n \n \tfor_each_ref(add_existing, &existing_refs);\n-\tfor (ref = transport_get_remote_refs(transport); ref; ref = ref->next) {\n+\tfor (ref = transport_get_remote_refs(transport, NULL); ref; ref = ref->next) {\n \t\tif (!starts_with(ref->name, \"refs/tags/\"))\n \t\t\tcontinue;\n \n@@ -336,7 +336,7 @@ static struct ref *get_ref_map(struct transport *transport,\n \t/* opportunistically-updated references: */\n \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n \n-\tconst struct ref *remote_refs = transport_get_remote_refs(transport);\n+\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n \n \tif (refspec_count) {\n \t\tstruct refspec *fetch_refspec;\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex c4be98ab9..c6e9847c5 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -96,7 +96,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (uploadpack != NULL)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n \n-\tref = transport_get_remote_refs(transport);\n+\tref = transport_get_remote_refs(transport, NULL);\n \tif (transport_disconnect(transport))\n \t\treturn 1;\n \ndiff --git a/builtin/remote.c b/builtin/remote.c\nindex d95bf904c..d0b6ff6e2 100644\n--- a/builtin/remote.c\n+++ b/builtin/remote.c\n@@ -862,7 +862,7 @@ static int get_remote_ref_states(const char *name,\n \tif (query) {\n \t\ttransport = transport_get(states->remote, states->remote->url_nr > 0 ?\n \t\t\tstates->remote->url[0] : NULL);\n-\t\tremote_refs = transport_get_remote_refs(transport);\n+\t\tremote_refs = transport_get_remote_refs(transport, NULL);\n \t\ttransport_disconnect(transport);\n \n \t\tstates->queried = 1;\ndiff --git a/transport.c b/transport.c\nindex c54a44630..dfc603b36 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1136,10 +1136,13 @@ int transport_push(struct transport *transport,\n \treturn 1;\n }\n \n-const struct ref *transport_get_remote_refs(struct transport *transport)\n+const struct ref *transport_get_remote_refs(struct transport *transport,\n+\t\t\t\t\t const struct argv_array *ref_patterns)\n {\n \tif (!transport->got_remote_refs) {\n-\t\ttransport->remote_refs = transport->vtable->get_refs_list(transport, 0, NULL);\n+\t\ttransport->remote_refs =\n+\t\t\ttransport->vtable->get_refs_list(transport, 0,\n+\t\t\t\t\t\t\t ref_patterns);\n \t\ttransport->got_remote_refs = 1;\n \t}\n \ndiff --git a/transport.h b/transport.h\nindex 731c78b67..daea4770c 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -178,7 +178,17 @@ int transport_push(struct transport *connection,\n \t\t int refspec_nr, const char **refspec, int flags,\n \t\t unsigned int * reject_reasons);\n \n-const struct ref *transport_get_remote_refs(struct transport *transport);\n+/*\n+ * Retrieve refs from a remote.\n+ *\n+ * Optionally a list of ref patterns can be provided which can be sent to the\n+ * server (when communicating using protocol v2) to enable it to limit the ref\n+ * advertisement. Since ref filtering is done on the server's end (and only\n+ * when using protocol v2), this can return refs which don't match the provided\n+ * ref_patterns.\n+ */\n+const struct ref *transport_get_remote_refs(struct transport *transport,\n+\t\t\t\t\t const struct argv_array *ref_patterns);\n \n int transport_fetch_refs(struct transport *transport, struct ref *refs);\n void transport_unlock_pack(struct transport *transport);\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340675","messageId":"20180228232252.102167-18-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 17/35] ls-remote: pass ref patterns when requesting a remote's refs","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:34Z","receivedAt":"2018-02-28T23:25:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct an argv_array of the ref patterns supplied via the command\nline and pass them to 'transport_get_remote_refs()' to be used when\ncommunicating protocol v2 so that the server can limit the ref\nadvertisement based on the supplied patterns.\n\nSigned-off-by: Brandon Williams \n---\n builtin/ls-remote.c | 12 ++++++++++--\n refs.c | 14 ++++++++++++++\n refs.h | 7 +++++++\n t/t5702-protocol-v2.sh | 26 ++++++++++++++++++++++++++\n 4 files changed, 57 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\nindex c6e9847c5..083ba8b29 100644\n--- a/builtin/ls-remote.c\n+++ b/builtin/ls-remote.c\n@@ -2,6 +2,7 @@\n #include \"cache.h\"\n #include \"transport.h\"\n #include \"remote.h\"\n+#include \"refs.h\"\n \n static const char * const ls_remote_usage[] = {\n \tN_(\"git ls-remote [--heads] [--tags] [--refs] [--upload-pack=]\\n\"\n@@ -43,6 +44,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tint show_symref_target = 0;\n \tconst char *uploadpack = NULL;\n \tconst char **pattern = NULL;\n+\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n \n \tstruct remote *remote;\n \tstruct transport *transport;\n@@ -74,8 +76,14 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (argc > 1) {\n \t\tint i;\n \t\tpattern = xcalloc(argc, sizeof(const char *));\n-\t\tfor (i = 1; i < argc; i++)\n+\t\tfor (i = 1; i < argc; i++) {\n \t\t\tpattern[i - 1] = xstrfmt(\"*/%s\", argv[i]);\n+\n+\t\t\tif (strchr(argv[i], '*'))\n+\t\t\t\targv_array_push(&ref_patterns, argv[i]);\n+\t\t\telse\n+\t\t\t\texpand_ref_pattern(&ref_patterns, argv[i]);\n+\t\t}\n \t}\n \n \tremote = remote_get(dest);\n@@ -96,7 +104,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n \tif (uploadpack != NULL)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n \n-\tref = transport_get_remote_refs(transport, NULL);\n+\tref = transport_get_remote_refs(transport, &ref_patterns);\n \tif (transport_disconnect(transport))\n \t\treturn 1;\n \ndiff --git a/refs.c b/refs.c\nindex 20ba82b43..58e9f88fb 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -13,6 +13,7 @@\n #include \"tag.h\"\n #include \"submodule.h\"\n #include \"worktree.h\"\n+#include \"argv-array.h\"\n \n /*\n * List of all available backends\n@@ -501,6 +502,19 @@ int refname_match(const char *abbrev_name, const char *full_name)\n \treturn 0;\n }\n \n+/*\n+ * Given a 'pattern' expand it by the rules in 'ref_rev_parse_rules' and add\n+ * the results to 'patterns'\n+ */\n+void expand_ref_pattern(struct argv_array *patterns, const char *pattern)\n+{\n+\tconst char **p;\n+\tfor (p = ref_rev_parse_rules; *p; p++) {\n+\t\tint len = strlen(pattern);\n+\t\targv_array_pushf(patterns, *p, len, pattern);\n+\t}\n+}\n+\n /*\n * *string and *len will only be substituted, and *string returned (for\n * later free()ing) if the string passed in is a magic short-hand form\ndiff --git a/refs.h b/refs.h\nindex 01be5ae32..292ca35ce 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -139,6 +139,13 @@ int resolve_gitlink_ref(const char *submodule, const char *refname,\n */\n int refname_match(const char *abbrev_name, const char *full_name);\n \n+/*\n+ * Given a 'pattern' expand it by the rules in 'ref_rev_parse_rules' and add\n+ * the results to 'patterns'\n+ */\n+struct argv_array;\n+void expand_ref_pattern(struct argv_array *patterns, const char *pattern);\n+\n int expand_ref(const char *str, int len, struct object_id *oid, char **ref);\n int dwim_ref(const char *str, int len, struct object_id *oid, char **ref);\n int dwim_log(const char *str, int len, struct object_id *oid, char **ref);\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex dc5f813be..562610fd2 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -32,6 +32,19 @@ test_expect_success 'list refs with git:// using protocol v2' '\n \ttest_cmp actual expect\n '\n \n+test_expect_success 'ref advertisment is filtered with ls-remote using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -c protocol.version=2 \\\n+\t\tls-remote \"$GIT_DAEMON_URL/parent\" master >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\t$(git -C \"$daemon_parent\" rev-parse refs/heads/master)$(printf \"\\t\")refs/heads/master\n+\tEOF\n+\n+\ttest_cmp actual expect\n+'\n+\n stop_git_daemon\n \n # Test protocol v2 with 'file://' transport\n@@ -54,4 +67,17 @@ test_expect_success 'list refs with file:// using protocol v2' '\n \ttest_cmp actual expect\n '\n \n+test_expect_success 'ref advertisment is filtered with ls-remote using protocol v2' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -c protocol.version=2 \\\n+\t\tls-remote \"file://$(pwd)/file_parent\" master >actual &&\n+\n+\tcat >expect <<-EOF &&\n+\t$(git -C file_parent rev-parse refs/heads/master)$(printf \"\\t\")refs/heads/master\n+\tEOF\n+\n+\ttest_cmp actual expect\n+'\n+\n test_done\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340676","messageId":"20180228232252.102167-19-bmwill@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"[PATCH v4 18/35] fetch: pass ref patterns when fetching","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-02-28T23:22:35Z","receivedAt":"2018-02-28T23:25:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Construct a list of ref patterns to be passed to\n'transport_get_remote_refs()' from the refspec to be used during the\nfetch. This list of ref patterns will be used to allow the server to\nfilter the ref advertisement when communicating using protocol v2.\n\nSigned-off-by: Brandon Williams \n---\n builtin/fetch.c | 16 +++++++++++++++-\n 1 file changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 850382f55..695fafe06 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -332,11 +332,25 @@ static struct ref *get_ref_map(struct transport *transport,\n \tstruct ref *rm;\n \tstruct ref *ref_map = NULL;\n \tstruct ref **tail = &ref_map;\n+\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n \n \t/* opportunistically-updated references: */\n \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n \n-\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n+\tconst struct ref *remote_refs;\n+\n+\tfor (i = 0; i < refspec_count; i++) {\n+\t\tif (!refspecs[i].exact_sha1) {\n+\t\t\tif (refspecs[i].pattern)\n+\t\t\t\targv_array_push(&ref_patterns, refspecs[i].src);\n+\t\t\telse\n+\t\t\t\texpand_ref_pattern(&ref_patterns, refspecs[i].src);\n+\t\t}\n+\t}\n+\n+\tremote_refs = transport_get_remote_refs(transport, &ref_patterns);\n+\n+\targv_array_clear(&ref_patterns);\n \n \tif (refspec_count) {\n \t\tstruct refspec *fetch_refspec;\n-- \n2.16.2.395.g2e18187dfd-goog\n\n"},{"id":"340745","messageId":"xmqqlgfbhc4t.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-1-bmwill@google.com","subject":"Re: [PATCH v4 00/35] protocol version 2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T18:41:22Z","receivedAt":"2018-03-01T18:41:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> Lots of changes since v3 (well more than between v2 and v3). Thanks for\n> all of the reviews on the last round, the series is getting more\n> polished.\n>\n> * Eliminated the \"# service\" line from the response from an HTTP\n> server. This means that the response to a v2 request is exactly the\n> same regardless of which transport you use! Docs for this have been\n> added as well.\n> * Changed how ref-patterns work with the `ls-refs` command. Instead of\n> using wildmatch all patterns must either match exactly or they can\n> contain a single '*' character at the end to mean that the prefix\n> must match. Docs for this have also been added.\n> * Lots of updates to the docs. Including documenting the\n> `stateless-connect` remote-helper command used by remote-curl to\n> handle the http transport.\n> * Fixed a number of bugs with the `fetch` command, one of which didn't\n> use objects from configured alternates.\n\nI noticed that this round is built on top of v2.16.0-rc0. It\ncertainly makes it easier to compare against the previous round\nwhich was built on top of that old commit and it is very much\nappreciated that a reroll does not involve pointless rebases.\n\nFor those who are helping from sidelines, it may be ehlpful to\nmention where in the history this was developed on, though, as\napplying these on the current 'master' has a handful of small\nconflicts.\n\nThanks, will replace and will comment on individual patches as\nneeded.\n"},{"id":"340750","messageId":"20180301191629.GA241240@google.com","threadId":"47524","inReplyTo":"xmqqlgfbhc4t.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 00/35] protocol version 2","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-01T19:16:29Z","receivedAt":"2018-03-01T19:16:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/01, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > Lots of changes since v3 (well more than between v2 and v3). Thanks for\n> > all of the reviews on the last round, the series is getting more\n> > polished.\n> >\n> > * Eliminated the \"# service\" line from the response from an HTTP\n> > server. This means that the response to a v2 request is exactly the\n> > same regardless of which transport you use! Docs for this have been\n> > added as well.\n> > * Changed how ref-patterns work with the `ls-refs` command. Instead of\n> > using wildmatch all patterns must either match exactly or they can\n> > contain a single '*' character at the end to mean that the prefix\n> > must match. Docs for this have also been added.\n> > * Lots of updates to the docs. Including documenting the\n> > `stateless-connect` remote-helper command used by remote-curl to\n> > handle the http transport.\n> > * Fixed a number of bugs with the `fetch` command, one of which didn't\n> > use objects from configured alternates.\n> \n> I noticed that this round is built on top of v2.16.0-rc0. It\n> certainly makes it easier to compare against the previous round\n> which was built on top of that old commit and it is very much\n> appreciated that a reroll does not involve pointless rebases.\n> \n> For those who are helping from sidelines, it may be ehlpful to\n> mention where in the history this was developed on, though, as\n> applying these on the current 'master' has a handful of small\n> conflicts.\n> \n> Thanks, will replace and will comment on individual patches as\n> needed.\n\nI've tried to keep building on the same base that I started with when\nsending out a new version of series, mostly because I thought it was\neasier to see what was different between rounds.\n\nI can, in the future, try to remember to put the commit its based on.\nDo we have any sort of guidance about the best practice here?\n\n-- \nBrandon Williams\n"},{"id":"340759","messageId":"xmqqk1uvfro3.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-3-bmwill@google.com","subject":"Re: [PATCH v4 02/35] pkt-line: allow peeking a packet line without consuming it","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T20:48:44Z","receivedAt":"2018-03-01T20:48:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> +enum packet_read_status packet_reader_read(struct packet_reader *reader)\n> +{\n> +\tif (reader->line_peeked) {\n> +\t\treader->line_peeked = 0;\n> +\t\treturn reader->status;\n> +\t}\n> +\n> +\treader->status = packet_read_with_status(reader->fd,\n> +\t\t\t\t\t\t &reader->src_buffer,\n> +\t\t\t\t\t\t &reader->src_len,\n> +\t\t\t\t\t\t reader->buffer,\n> +\t\t\t\t\t\t reader->buffer_size,\n> +\t\t\t\t\t\t &reader->pktlen,\n> +\t\t\t\t\t\t reader->options);\n> +\n> +\tswitch (reader->status) {\n> +\tcase PACKET_READ_EOF:\n> +\t\treader->pktlen = -1;\n> +\t\treader->line = NULL;\n> +\t\tbreak;\n> +\tcase PACKET_READ_NORMAL:\n> +\t\treader->line = reader->buffer;\n> +\t\tbreak;\n> +\tcase PACKET_READ_FLUSH:\n> +\t\treader->pktlen = 0;\n> +\t\treader->line = NULL;\n> +\t\tbreak;\n> +\t}\n> +\n> +\treturn reader->status;\n> +}\n\nWith the way _peek() interface interacts with the reader instance\n(which by the way I find is well designed), it is understandable\nthat we want almost everything available in reader's fields, but\nhaving to manually clear pktlen field upon non NORMAL status feels\na bit strange. \n\nPerhaps that is because the underlying packet_read_with_status()\ndoes not set *pktlen in these cases? Shouldn't it be doing that so\nthe caller does not have to?\n\nA similar comment applies for reader's line field. In priniciple,\nas the status field is part of a reader, it does not have to exist\nas a separate field, i.e.\n\n\t#define line_of(reader) \\\n\t\t((reader).status == PACKET_READ_NORMAL ? \\\n\t\t(reader).buffer : NULL)\n\ncan be used to as substitute for it. I guess it depends on how the\nactual callers wants to use this interface.\n"},{"id":"340760","messageId":"xmqqfu5jfrlb.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-4-bmwill@google.com","subject":"Re: [PATCH v4 03/35] pkt-line: add delim packet support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T20:50:24Z","receivedAt":"2018-03-01T20:51:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> One of the design goals of protocol-v2 is to improve the semantics of\n> flush packets. Currently in protocol-v1, flush packets are used both to\n> indicate a break in a list of packet lines as well as an indication that\n> one side has finished speaking. This makes it particularly difficult\n> to implement proxies as a proxy would need to completely understand git\n> protocol instead of simply looking for a flush packet.\n\nGood ;-) Yes, this has been one of the largest gripe about the\nsmart-http support code we have.\n"},{"id":"340761","messageId":"xmqqbmg7fr5s.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180301191629.GA241240@google.com","subject":"Re: [PATCH v4 00/35] protocol version 2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T20:59:43Z","receivedAt":"2018-03-01T20:59:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> I've tried to keep building on the same base that I started with when\n> sending out a new version of series, mostly because I thought it was\n> easier to see what was different between rounds.\n\nYes. It indeed is easier to see the evolution if the series does\nnot get rebased needlessly.\n\n> I can, in the future, try to remember to put the commit its based on.\n> Do we have any sort of guidance about the best practice here?\n\nI recall we taught a new \"--base\" option to \"format-patch\" not too\nlong ago, so one way to do so may be:\n\n $ git format-patch --cover-letter --base=v2.16.0-rc0 master..bw/protocol-v2\n $ tail -4 0000-cover*.txt\n base-commit: 1eaabe34fc6f486367a176207420378f587d3b48\n --\n 2.16.2-345-g7e31236f65\n\nperhaps?\n"},{"id":"340762","messageId":"xmqq7eqvfqx1.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"xmqqfu5jfrlb.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/35] pkt-line: add delim packet support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T21:04:58Z","receivedAt":"2018-03-01T21:05:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Brandon Williams writes:\n>\n>> One of the design goals of protocol-v2 is to improve the semantics of\n>> flush packets. Currently in protocol-v1, flush packets are used both to\n>> indicate a break in a list of packet lines as well as an indication that\n>> one side has finished speaking. This makes it particularly difficult\n>> to implement proxies as a proxy would need to completely understand git\n>> protocol instead of simply looking for a flush packet.\n>\n> Good ;-) Yes, this has been one of the largest gripe about the\n> smart-http support code we have.\n\nHmph, strictly speaking, the \"delim\" does not have to be a part of\nhow packetized stream is defined. As long as we stop abusing flush\nas \"This is merely an end of one segment of what I say.\" and make it\nalways mean \"I am done speaking, it is your turn.\", the application\npayload can define its own syntax to separate groups of packets.\n\nI do not mind having this \"delim\" thing defined at the protocol\nlevel too much, though.\n"},{"id":"340763","messageId":"xmqq371jfpzj.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-6-bmwill@google.com","subject":"Re: [PATCH v4 05/35] upload-pack: factor out processing lines","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T21:25:04Z","receivedAt":"2018-03-01T21:25:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> Factor out the logic for processing shallow, deepen, deepen_since, and\n> deepen_not lines into their own functions to simplify the\n> 'receive_needs()' function in addition to making it easier to reuse some\n> of this logic when implementing protocol_v2.\n\nThese little functions that still require their incoming data to\nbegin with fixed prefixes feels a bit strange way to refactor the\nlogic for later reuse (when I imagine \"reuse\", the first use case\nthat comes to my mind is \"this data source our new code reads from\ngives the same data as the old 'shallow' packet used to give, but in\na different syntax\"---so I'd restructure the code in such a way that\nthe caller figures out the syntax part and the called helper just\ngroks the \"information contents\" unwrapped from the surface syntax;\nthe syntax may be different in the new codepath but once unwrapped,\nthe \"information contents\" to be processed would not be different\nhence we can reuse the helper).\n\nIOW, I would have expected the caller to be not like this:\n\n> -\t\tif (skip_prefix(line, \"shallow \", &arg)) {\n> -\t\t\tstruct object_id oid;\n> -\t\t\tstruct object *object;\n> -\t\t\tif (get_oid_hex(arg, &oid))\n> -\t\t\t\tdie(\"invalid shallow line: %s\", line);\n> -\t\t\tobject = parse_object(&oid);\n> -\t\t\tif (!object)\n> -\t\t\t\tcontinue;\n> -\t\t\tif (object->type != OBJ_COMMIT)\n> -\t\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n> -\t\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n> -\t\t\t\tobject->flags |= CLIENT_SHALLOW;\n> -\t\t\t\tadd_object_array(object, NULL, &shallows);\n> -\t\t\t}\n> +\t\tif (process_shallow(line, &shallows))\n> \t\t\tcontinue;\n> +\t\tif (process_deepen(line, &depth))\n> \t\t\tcontinue;\n\t\t...\n\nbut more like\n\n\t\tif (skip_prefix(line, \"shallow \", &arg) {\n\t\t\tprocess_shallow(arg, &shallows);\n\t\t\tcontinue;\n\t\t}\n\t\tif (skip_prefix(line, \"deepen \", &arg) {\n\t\t\tprocess_deepen(arg, &depth);\n\t\t\tcontinue;\n\t\t}\n\t\t...\n\nI need to defer the final judgment until I see how they are used,\nthough. It's not too big a deal either way---it just felt \"not\nquite right\" to me.\n\n\n"},{"id":"340764","messageId":"xmqqy3jbebeb.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-7-bmwill@google.com","subject":"Re: [PATCH v4 06/35] transport: use get_refs_via_connect to get refs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T21:25:32Z","receivedAt":"2018-03-01T21:25:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> Remove code duplication and use the existing 'get_refs_via_connect()'\n> function to retrieve a remote's heads in 'fetch_refs_via_pack()' and\n> 'git_transport_push()'.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> transport.c | 18 ++++--------------\n> 1 file changed, 4 insertions(+), 14 deletions(-)\n\nNice ;-)\n"},{"id":"340767","messageId":"20180301224916.GB241240@google.com","threadId":"47524","inReplyTo":"xmqq7eqvfqx1.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/35] pkt-line: add delim packet support","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-01T22:49:16Z","receivedAt":"2018-03-01T22:49:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/01, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > Brandon Williams writes:\n> >\n> >> One of the design goals of protocol-v2 is to improve the semantics of\n> >> flush packets. Currently in protocol-v1, flush packets are used both to\n> >> indicate a break in a list of packet lines as well as an indication that\n> >> one side has finished speaking. This makes it particularly difficult\n> >> to implement proxies as a proxy would need to completely understand git\n> >> protocol instead of simply looking for a flush packet.\n> >\n> > Good ;-) Yes, this has been one of the largest gripe about the\n> > smart-http support code we have.\n> \n> Hmph, strictly speaking, the \"delim\" does not have to be a part of\n> how packetized stream is defined. As long as we stop abusing flush\n> as \"This is merely an end of one segment of what I say.\" and make it\n> always mean \"I am done speaking, it is your turn.\", the application\n> payload can define its own syntax to separate groups of packets.\n\nThanks actually a good point. We could just as easily have the delim\npacket to be an empty packet-line \"0004\" or something like that.\n\n> \n> I do not mind having this \"delim\" thing defined at the protocol\n> level too much, though.\n\n-- \nBrandon Williams\n"},{"id":"340768","messageId":"xmqqinafe6ih.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-13-bmwill@google.com","subject":"Re: [PATCH v4 12/35] serve: introduce git-serve","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T23:11:02Z","receivedAt":"2018-03-01T23:11:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> Documentation/technical/protocol-v2.txt | 171 ++++++++++++++++\n\nUnlike other things in Documentation/technical/, this is not listed\non TECH_DOCS list in Documentation/Makefile. Shouldn't it be?\n"},{"id":"340769","messageId":"xmqqefl3e505.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180301224916.GB241240@google.com","subject":"Re: [PATCH v4 03/35] pkt-line: add delim packet support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-01T23:43:38Z","receivedAt":"2018-03-01T23:43:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> On 03/01, Junio C Hamano wrote:\n> ...\n>> Hmph, strictly speaking, the \"delim\" does not have to be a part of\n>> how packetized stream is defined. As long as we stop abusing flush\n>> as \"This is merely an end of one segment of what I say.\" and make it\n>> always mean \"I am done speaking, it is your turn.\", the application\n>> payload can define its own syntax to separate groups of packets.\n>\n> Thanks actually a good point. We could just as easily have the delim\n> packet to be an empty packet-line \"0004\" or something like that.\n\nYes. As long as there is an easy and obvious \"cannot be a value\"\nconstant, you can use it as a delimiter defined at the application\nlevel. For example, your command-request uses delim, like so:\n\n+ request = empty-request | command-request\n+ empty-request = flush-pkt\n+ command-request = command\n+\t\t capability-list\n+\t\t (command-args)\n+\t\t flush-pkt\n+ command = PKT-LINE(\"command=\" key LF)\n+ command-args = delim-pkt\n+\t\t *PKT-Line(arg LF)\n\nto mark the end of cap list, but if an empty packet does not make\nsense as a member of a cap list and a commmand args list, then an\nempty packet between cap list and command arg can be used instead.\nA protocol-ignorant proxy can still work just fine.\n\nHaving a defined delim at the protocol level is often convenient, of\ncourse, but once the application starts calling for multi-level\ndelimiters (i.e. maybe there are chapters and sections inside each\nchapter in a single request message), it would not be sufficient to\ndefine a single delim packet type. The application layer needs to\ndefine its own convention (e.g. if no \"empty\" section is allowed,\nthen \"two consecutive delim is a chapter break; one delim is a\nsection break\" can become a viable way to emulate multi-level\ndelimiters).\n"},{"id":"340832","messageId":"nycvar.QRO.7.76.6.1803022106030.56@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","threadId":"47524","inReplyTo":"20180228232252.102167-35-bmwill@google.com","subject":"Re: [PATCH v4 34/35] remote-curl: implement stateless-connect command","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-03-02T20:07:52Z","receivedAt":"2018-03-02T20:08:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brandon,\n\nOn Wed, 28 Feb 2018, Brandon Williams wrote:\n\n> diff --git a/remote-curl.c b/remote-curl.c\n> index 66a53f74b..3f882d766 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -188,7 +188,12 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n> [...]\n> +static size_t proxy_in(char *buffer, size_t eltsize,\n> +\t\t size_t nmemb, void *userdata)\n> +{\n> +\tsize_t max;\n> +\tstruct proxy_state *p = userdata;\n> +\tsize_t avail = p->request_buffer.len - p->pos;\n> +\n> +\n> +\tif (eltsize != 1)\n> +\t\tBUG(\"curl read callback called with size = %zu != 1\", eltsize);\n\nThe format specified %z is not actually portable. Please use PRIuMAX and\ncast to (uintmax_t) instead.\n\nThis breaks the Windows build of `pu` (before that, there was still a test\nfailure that I did not have the time to chase down).\n\nCiao,\nDscho\n"},{"id":"340836","messageId":"xmqq4llyciq8.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-13-bmwill@google.com","subject":"Re: [PATCH v4 12/35] serve: introduce git-serve","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-02T20:42:23Z","receivedAt":"2018-03-02T20:42:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> + Capabilities\n> +~~~~~~~~~~~~~~\n> +\n> +There are two different types of capabilities: normal capabilities,\n> +which can be used to to convey information or alter the behavior of a\n> +request, and commands, which are the core actions that a client wants to\n> +perform (fetch, push, etc).\n> +\n> +All commands must only last a single round and be stateless from the\n> +perspective of the server side. All state MUST be retained and managed\n> +by the client process. This permits simple round-robin load-balancing\n> +on the server side, without needing to worry about state management.\n> +\n> +Clients MUST NOT require state management on the server side in order to\n> +function correctly.\n\nThis somehow feels a bit too HTTP centric worldview that potentially\nmay penalize those who do not mind stateful services.\n\n> + agent\n> +-------\n> +\n> +The server can advertise the `agent` capability with a value `X` (in the\n> +form `agent=X`) to notify the client that the server is running version\n> +`X`. The client may optionally send its own agent string by including\n> +the `agent` capability with a value `Y` (in the form `agent=Y`) in its\n> +request to the server (but it MUST NOT do so if the server did not\n> +advertise the agent capability).\n\nAre there different degrees of permissiveness between \"The server\nCAN\" and \"The client MAY\" above, or is the above paragraph merely\nbeing fuzzy?\n\nI notice that, with the above \"MUST NOT\", it is impossible for a\nserver to collect voluntary census information from client without\nrevealing its own \"version\". Because in principle it is not\nsensible to allow one side to send random capabilities without first\nmaking sure that the other side understands them, unsolicited\n\"agent\" from the client over a channel where the server did not say\nit would accept one is quite fine, and the server can always say\nsomething silly like \"agent=undisclosed\" to allow the clients to\nvolunteer their own version, but the definition of this capability\nsmells like conflating two unrelated things (i.e. advertising your\nown version vs permission to announce yourself).\n\n"},{"id":"340837","messageId":"xmqqzi3qb3if.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-13-bmwill@google.com","subject":"Re: [PATCH v4 12/35] serve: introduce git-serve","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-02T20:56:24Z","receivedAt":"2018-03-02T20:56:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> +\t/*\n> +\t * Function queried to see if a capability should be advertised.\n> +\t * Optionally a value can be specified by adding it to 'value'.\n> +\t * If a value is added to 'value', the server will advertise this\n> +\t * capability as \"=\" instead of \"\".\n> +\t */\n> +\tint (*advertise)(struct repository *r, struct strbuf *value);\n\nSo this is \"do we tell them about this capability?\"\n\n> +static void advertise_capabilities(void)\n> +{\n> +\t...\n> +\tfor (i = 0; i < ARRAY_SIZE(capabilities); i++) {\n> +\t\tstruct protocol_capability *c = &capabilities[i];\n> +\n> +\t\tif (c->advertise(the_repository, &value)) {\n> +\t\t\tstrbuf_addstr(&capability, c->name);\n> + ...\n\nAnd used as such in this function. We tell the other side about the\ncapability only when .advertise returns true.\n\n> +static int is_valid_capability(const char *key)\n> +{\n> +\tconst struct protocol_capability *c = get_capability(key);\n> +\n> +\treturn c && c->advertise(the_repository, NULL);\n> +}\n\nBut this is different---the other side mentioned a capability's\nname, and we looked it up from our table to see if we know about it\n(i.e. NULL-ness of c), but in addition, we ask if we would tell them\nabout it if we were advertising. I am not sure how I should feel\nabout it (yet).\n\n> +static int is_command(const char *key, struct protocol_capability **command)\n> +{\n> +\tconst char *out;\n> +\n> +\tif (skip_prefix(key, \"command=\", &out)) {\n> +\t\tstruct protocol_capability *cmd = get_capability(out);\n> +\n> +\t\tif (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)\n> +\t\t\tdie(\"invalid command '%s'\", out);\n> +\t\tif (*command)\n> +\t\t\tdie(\"command already requested\");\n\nShouldn't these two checks that lead to die the other way around?\nWhen they give us \"command=frotz\" and we already have *command, it\nwould be an error whether we understand 'frotz' or not.\n\nWho are the target audience of these \"die\"? Are they meant to be\ncommunicated back to the other side of the connection, or are they\nonly to be sent to the \"server log\"?\n\nThe latter one may want to say what two conflicting commands are in\nthe log message, perhaps?\n\n> +\t\t*command = cmd;\n\n"},{"id":"340838","messageId":"xmqqmuzqb2ph.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-14-bmwill@google.com","subject":"Re: [PATCH v4 13/35] ls-refs: introduce ls-refs server command","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-02T21:13:46Z","receivedAt":"2018-03-02T21:13:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> + ls-refs\n> +---------\n> +\n> +`ls-refs` is the command used to request a reference advertisement in v2.\n> +Unlike the current reference advertisement, ls-refs takes in arguments\n> +which can be used to limit the refs sent from the server.\n\nOK.\n\n> +Additional features not supported in the base command will be advertised\n> +as the value of the command in the capability advertisement in the form\n> +of a space separated list of features, e.g. \"=\n> +\".\n\nDoesn't this explain the general convention that applies to any\ncommand, not just ls-refs command? As a part of ls-refs section,\n in the above explanation is always a constant \"ls-refs\",\nright?\n\nIt is a bit unclear how in the above description are\nrelated to \"arguments\" in the following paragraph. Do the server\nthat can show symref and peeled tags and that can limit the output\nwith ref-pattern advertise these three as supported features, i.e.\n\n\tls-refs=symrefs peel ref-pattern\n\nor something? Would there a case where a \"feature\" does not\ncorrespond 1:1 to an argument to the command, and if so how would\nthe server and the client negotiate use of such a feature?\n\n> + ref-pattern \n> +\tWhen specified, only references matching one of the provided\n> +\tpatterns are displayed. A pattern is either a valid refname\n> +\t(e.g. refs/heads/master), in which a ref must match the pattern\n> +\texactly, or a prefix of a ref followed by a single '*' wildcard\n> +\tcharacter (e.g. refs/heads/*), in which a ref must have a prefix\n> +\tequal to the pattern up to the wildcard character.\n\nI thought the recent concensus was left-anchored prefix match that\nhonors /-directory boundary, i.e. no explicit asterisk and just\nsaying \"refs/heads\" is enough to match \"refs/heads\" itself and\n\"refs/heads/master\" but not \"refs/headscarf\"?\n"},{"id":"340840","messageId":"xmqq8tbaazx9.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-18-bmwill@google.com","subject":"Re: [PATCH v4 17/35] ls-remote: pass ref patterns when requesting a remote's refs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-02T22:13:54Z","receivedAt":"2018-03-02T22:14:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> Construct an argv_array of the ref patterns supplied via the command\n> line and pass them to 'transport_get_remote_refs()' to be used when\n> communicating protocol v2 so that the server can limit the ref\n> advertisement based on the supplied patterns.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> builtin/ls-remote.c | 12 ++++++++++--\n> refs.c | 14 ++++++++++++++\n> refs.h | 7 +++++++\n> t/t5702-protocol-v2.sh | 26 ++++++++++++++++++++++++++\n> 4 files changed, 57 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/ls-remote.c b/builtin/ls-remote.c\n> index c6e9847c5..083ba8b29 100644\n> --- a/builtin/ls-remote.c\n> +++ b/builtin/ls-remote.c\n> @@ -2,6 +2,7 @@\n> #include \"cache.h\"\n> #include \"transport.h\"\n> #include \"remote.h\"\n> +#include \"refs.h\"\n> \n> static const char * const ls_remote_usage[] = {\n> \tN_(\"git ls-remote [--heads] [--tags] [--refs] [--upload-pack=]\\n\"\n> @@ -43,6 +44,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n> \tint show_symref_target = 0;\n> \tconst char *uploadpack = NULL;\n> \tconst char **pattern = NULL;\n> +\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n> \n> \tstruct remote *remote;\n> \tstruct transport *transport;\n> @@ -74,8 +76,14 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n> \tif (argc > 1) {\n> \t\tint i;\n> \t\tpattern = xcalloc(argc, sizeof(const char *));\n> -\t\tfor (i = 1; i < argc; i++)\n> +\t\tfor (i = 1; i < argc; i++) {\n> \t\t\tpattern[i - 1] = xstrfmt(\"*/%s\", argv[i]);\n> +\n> +\t\t\tif (strchr(argv[i], '*'))\n> +\t\t\t\targv_array_push(&ref_patterns, argv[i]);\n> +\t\t\telse\n> +\t\t\t\texpand_ref_pattern(&ref_patterns, argv[i]);\n> +\t\t}\n> \t}\n> \n> \tremote = remote_get(dest);\n> @@ -96,7 +104,7 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)\n> \tif (uploadpack != NULL)\n> \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);\n> \n> -\tref = transport_get_remote_refs(transport, NULL);\n> +\tref = transport_get_remote_refs(transport, &ref_patterns);\n\nYup, this is a logical and an obvious conclusion of the past handful\nof steps ;-) I actually was wondering why the previous step didn't\ndo this already, but the resulting series is easier to understand if\nthis is kept as a separate step.\n\nHowever, this also means that traditional pattern language ls-remote\nused to support dictates what ls-refs command over the wire can\ntake, which may not be optimal.\n"},{"id":"340841","messageId":"xmqq4llyazma.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-19-bmwill@google.com","subject":"Re: [PATCH v4 18/35] fetch: pass ref patterns when fetching","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-02T22:20:29Z","receivedAt":"2018-03-02T22:20:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 850382f55..695fafe06 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -332,11 +332,25 @@ static struct ref *get_ref_map(struct transport *transport,\n> \tstruct ref *rm;\n> \tstruct ref *ref_map = NULL;\n> \tstruct ref **tail = &ref_map;\n> +\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n> \n> \t/* opportunistically-updated references: */\n> \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n> \n> -\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n> +\tconst struct ref *remote_refs;\n> +\n> +\tfor (i = 0; i < refspec_count; i++) {\n> +\t\tif (!refspecs[i].exact_sha1) {\n> +\t\t\tif (refspecs[i].pattern)\n> +\t\t\t\targv_array_push(&ref_patterns, refspecs[i].src);\n> +\t\t\telse\n> +\t\t\t\texpand_ref_pattern(&ref_patterns, refspecs[i].src);\n> +\t\t}\n> +\t}\n> +\n> +\tremote_refs = transport_get_remote_refs(transport, &ref_patterns);\n> +\n> +\targv_array_clear(&ref_patterns);\n\nIs the idea here, which is shared with 17/35 about ls-remote, that\nwe used to grab literally everything they have in remote_refs, but\nwe have code in place to filter that set using refspecs given in the\nremote.*.fetch configuration, so it is OK as long as we grab everything\nthat would match the remote.*.fetch pattern? That is, grabbing too\nmuch is acceptable, but if we populated ref_patterns[] with too few\npatterns and fail to ask refs that would match our refspec it would\nbe a bug?\n\nThe reason behind this question is that I am wondering if/how we can\ntake advantage of this remote-side pre-filtering while doing \"fetch\n--prune\".\n\nThanks.\n\n> \n> \tif (refspec_count) {\n> \t\tstruct refspec *fetch_refspec;\n"},{"id":"340842","messageId":"xmqqzi3q9ku6.fsf@gitster-ct.c.googlers.com","threadId":"47524","inReplyTo":"20180228232252.102167-20-bmwill@google.com","subject":"Re: [PATCH v4 19/35] push: pass ref patterns when pushing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-03-02T22:25:05Z","receivedAt":"2018-03-02T22:25:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> Construct a list of ref patterns to be passed to 'get_refs_list()' from\n> the refspec to be used during the push. This list of ref patterns will\n> be used to allow the server to filter the ref advertisement when\n> communicating using protocol v2.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> transport.c | 26 +++++++++++++++++++++++++-\n> 1 file changed, 25 insertions(+), 1 deletion(-)\n\nWhen you are pushing 'master', we no longer hear what the other end\nhas at 'next', with this change, no?\n\nIn a project whose 'master' is extended primarily by merging topics\nthat have been cooking in 'next', old way of pushing would only have\ntransferred the merge commits and resulting trees but not bulk of\nthe blob data because they are all available on 'next', would it\nmake the object transfer far less efficient, I wonder?\n\nI guess it is OK only because the push side of the current protocol\ndoes not do common ancestor discovery exchange ;-)\n\n>\n> diff --git a/transport.c b/transport.c\n> index dfc603b36..bf7ba6879 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -1026,11 +1026,35 @@ int transport_push(struct transport *transport,\n> \t\tint porcelain = flags & TRANSPORT_PUSH_PORCELAIN;\n> \t\tint pretend = flags & TRANSPORT_PUSH_DRY_RUN;\n> \t\tint push_ret, ret, err;\n> +\t\tstruct refspec *tmp_rs;\n> +\t\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n> +\t\tint i;\n> \n> \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n> \t\t\treturn -1;\n> \n> -\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n> +\t\ttmp_rs = parse_push_refspec(refspec_nr, refspec);\n> +\t\tfor (i = 0; i < refspec_nr; i++) {\n> +\t\t\tconst char *pattern = NULL;\n> +\n> +\t\t\tif (tmp_rs[i].dst)\n> +\t\t\t\tpattern = tmp_rs[i].dst;\n> +\t\t\telse if (tmp_rs[i].src && !tmp_rs[i].exact_sha1)\n> +\t\t\t\tpattern = tmp_rs[i].src;\n> +\n> +\t\t\tif (pattern) {\n> +\t\t\t\tif (tmp_rs[i].pattern)\n> +\t\t\t\t\targv_array_push(&ref_patterns, pattern);\n> +\t\t\t\telse\n> +\t\t\t\t\texpand_ref_pattern(&ref_patterns, pattern);\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tremote_refs = transport->vtable->get_refs_list(transport, 1,\n> +\t\t\t\t\t\t\t &ref_patterns);\n> +\n> +\t\targv_array_clear(&ref_patterns);\n> +\t\tfree_refspec(refspec_nr, tmp_rs);\n> \n> \t\tif (flags & TRANSPORT_PUSH_ALL)\n> \t\t\tmatch_flags |= MATCH_REFS_ALL;\n"},{"id":"340865","messageId":"20180303042405.GA27689@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180223210904.GB234838@google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-03T04:24:05Z","receivedAt":"2018-03-03T04:24:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 23, 2018 at 01:09:04PM -0800, Brandon Williams wrote:\n\n> > By the way, any decision here would presumably need to be extended to\n> > git-serve, etc. The current property is that it's safe to fetch from an\n> > untrusted repository, even over ssh. If we're keeping that for protocol\n> > v1, we'd want it to apply to protocol v2, as well.\n> \n> This may be more complicated. Right now (for backward compatibility)\n> all fetches for v2 are issued to the upload-pack endpoint. So even\n> though I've introduced git-serve it doesn't have requests issued to it\n> and no requests can be issued to it currently (support isn't added to\n> http-backend or git-daemon). This just means that the command already\n> exists to make it easy for testing specific v2 stuff and if we want to\n> expose it as an endpoint (like when we have a brand new server command\n> that is completely incompatible with v1) its already there and support\n> just needs to be plumbed in.\n> \n> This whole notion of treating upload-pack differently from receive-pack\n> has bad consequences for v2 though. The idea for v2 is to be able to\n> run any number of commands via the same endpoint, so at the end of the\n> day the endpoint you used is irrelevant. So you could issue both fetch\n> and push commands via the same endpoint in v2 whether its git-serve,\n> receive-pack, or upload-pack. So really, like Jonathan has said\n> elsewhere, we need to figure out how to be ok with having receive-pack\n> and upload-pack builtins, or having neither of them builtins, because it\n> doesn't make much sense for v2 to straddle that line.\n\nIt seems like it would be OK if the whole code path of git-serve\ninvoking upload-pack happened without being a builtin, even if it would\nbe possible to run a builtin receive-pack from that same (non-builtin)\ngit-serve.\n\nRemember that the client is driving the whole operation here, and we can\nassume that git-serve is operating on the client's behalf. So a client\nwho chooses not to trigger receive-pack would be fine.\n\n-Peff\n"},{"id":"340866","messageId":"20180303042558.GB27689@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180223212231.GC234838@google.com","subject":"Re: [PATCH v3 11/35] test-pkt-line: introduce a packet-line test helper","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-03T04:25:59Z","receivedAt":"2018-03-03T04:26:05Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 23, 2018 at 01:22:31PM -0800, Brandon Williams wrote:\n\n> On 02/22, Stefan Beller wrote:\n> > On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> > \n> > > +static void pack_line(const char *line)\n> > > +{\n> > > + if (!strcmp(line, \"0000\") || !strcmp(line, \"0000\\n\"))\n> > \n> > From our in-office discussion:\n> > v1/v0 packs pktlines twice in http, which is not possible to\n> > construct using this test helper when using the same string\n> > for the packed and unpacked representation of flush and delim packets,\n> > i.e. test-pkt-line --pack $(test-pkt-line --pack 0000) would produce\n> > '0000' instead of '00090000\\n'.\n> > To fix it we'd have to replace the unpacked versions of these pkts to\n> > something else such as \"FLUSH\" \"DELIM\".\n> > \n> > However as we do not anticipate the test helper to be used in further\n> > tests for v0, this ought to be no big issue.\n> > Maybe someone else cares though?\n> \n> I'm going to punt and say, if someone cares enough they can update this\n> test-helper when they want to use it for v1/v0 stuff.\n\nI recently add packetize and depacketize helpers for testing v0 streams;\nsee 4414a15002 (t/lib-git-daemon: add network-protocol helpers,\n2018-01-24). Is it worth folding these together?\n\n-Peff\n"},{"id":"340867","messageId":"20180303043338.GC27689@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180223214557.GF234838@google.com","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-03T04:33:39Z","receivedAt":"2018-03-03T04:33:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 23, 2018 at 01:45:57PM -0800, Brandon Williams wrote:\n\n> I think this is the price of extending the protocol in a backward\n> compatible way. If we don't want to be backwards compatible (allowing\n> for graceful fallback to v1) then we could design this differently.\n> Even so we're not completely out of luck just yet.\n> \n> Back when I introduced the GIT_PROTOCOL side-channel I was able to\n> demonstrate that arbitrary data could be sent to the server and it would\n> only respect the stuff it knows about. This means that we can do a\n> follow up to v2 at some point to introduce an optimization where we can\n> stuff a request into GIT_PROTOCOL and short-circuit the first round-trip\n> if the server supports it.\n\nIf that's our end-game, it does make me wonder if we'd be happier just\njumping to that at first. Before you started the v2 protocol work, I had\na rough patch series passing what I called \"early capabilities\". The\nidea was to let the client speak a few optional capabilities before the\nref advertisement, and be ready for the server to ignore them\ncompletely. That doesn't clean up all the warts with the v0 protocol,\nbut it handles the major one (allowing more efficient ref\nadvertisements).\n\nI dunno. There's a lot more going on here in v2 and I'm not sure I've\nfully digested it.\n\n> The great thing about this is that from the POV of the git-client, it\n> doesn't care if its speaking using the git://, ssh://, file://, or\n> http:// transport; it's all the same protocol. In my next re-roll I'll\n> even drop the \"# service\" bit from the http server response and then the\n> responses will truly be identical in all cases.\n\nThis part has me a little confused still. The big difference between\nhttp and the other protocols is that the other ones are full-duplex, and\nhttp is a series of stateless request/response pairs.\n\nAre the other protocols becoming stateless request/response pairs, too?\nOr will they be \"the same protocol\" only in the sense of using the same\ntransport?\n\n(There are a lot of reasons not to like the stateless pair thing; it has\nsome horrid corner cases during want/have negotiation).\n\n-Peff\n"},{"id":"340868","messageId":"20180303044338.GD27689@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180228232252.102167-14-bmwill@google.com","subject":"Re: [PATCH v4 13/35] ls-refs: introduce ls-refs server command","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-03T04:43:38Z","receivedAt":"2018-03-03T04:43:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 28, 2018 at 03:22:30PM -0800, Brandon Williams wrote:\n\n> +static void add_pattern(struct pattern_list *patterns, const char *pattern)\n> +{\n> +\tstruct ref_pattern p;\n> +\tconst char *wildcard;\n> +\n> +\tp.pattern = strdup(pattern);\n\nxstrdup?\n\n> +\twildcard = strchr(pattern, '*');\n> +\tif (wildcard) {\n> +\t\tp.wildcard_pos = wildcard - pattern;\n> +\t} else {\n> +\t\tp.wildcard_pos = -1;\n> +\t}\n\nHmm, so this would accept stuff like \"refs/heads/*/foo\" but quietly\nignore the \"/foo\" part.\n\nIt also accepts \"refs/h*\" to get \"refs/heads\" and \"refs/hello\". I think\nit's worth going for the most-restrictive thing to start with, since\nthat enables a lot more server operations without worrying about\nbreaking compatibility.\n\n-Peff\n"},{"id":"341041","messageId":"20180305182155.GB72475@google.com","threadId":"47524","inReplyTo":"20180303044338.GD27689@sigill.intra.peff.net","subject":"Re: [PATCH v4 13/35] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-05T18:21:55Z","receivedAt":"2018-03-05T18:22:04Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/02, Jeff King wrote:\n> On Wed, Feb 28, 2018 at 03:22:30PM -0800, Brandon Williams wrote:\n> \n> > +static void add_pattern(struct pattern_list *patterns, const char *pattern)\n> > +{\n> > +\tstruct ref_pattern p;\n> > +\tconst char *wildcard;\n> > +\n> > +\tp.pattern = strdup(pattern);\n> \n> xstrdup?\n> \n> > +\twildcard = strchr(pattern, '*');\n> > +\tif (wildcard) {\n> > +\t\tp.wildcard_pos = wildcard - pattern;\n> > +\t} else {\n> > +\t\tp.wildcard_pos = -1;\n> > +\t}\n> \n> Hmm, so this would accept stuff like \"refs/heads/*/foo\" but quietly\n> ignore the \"/foo\" part.\n\nYeah that's true...this should probably not do that. Since\n\"refs/heads/*/foo\" violates what the spec is, really this should error\nout as an invalid pattern.\n\n> \n> It also accepts \"refs/h*\" to get \"refs/heads\" and \"refs/hello\". I think\n> it's worth going for the most-restrictive thing to start with, since\n> that enables a lot more server operations without worrying about\n> breaking compatibility.\n\nAnd just to clarify what do you see as being the most-restrictive case\nof patterns that would should use?\n\n-- \nBrandon Williams\n"},{"id":"341043","messageId":"20180305182914.GD28067@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180305182155.GB72475@google.com","subject":"Re: [PATCH v4 13/35] ls-refs: introduce ls-refs server command","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-03-05T18:29:14Z","receivedAt":"2018-03-05T18:29:25Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nOn Mon, Mar 05, 2018 at 10:21:55AM -0800, Brandon Williams wrote:\n> On 03/02, Jeff King wrote:\n\n>> It also accepts \"refs/h*\" to get \"refs/heads\" and \"refs/hello\". I think\n>> it's worth going for the most-restrictive thing to start with, since\n>> that enables a lot more server operations without worrying about\n>> breaking compatibility.\n>\n> And just to clarify what do you see as being the most-restrictive case\n> of patterns that would should use?\n\nPeff, can you say a little more about the downsides of accepting\nrefs/h*?\n\nIIRC the \"git push\" command already accepts such refspecs, so there's a\nbenefit to accepting them. Reftable and packed-refs support such\nqueries about as efficiently as refs/heads/*. For loose refs, readdir\ndoesn't provide a way to restrict which files you look at, but loose\nrefs are always slow anyway. :)\n\nIn other words, I see real benefits and I don't see much in the way of\ncosts, so I'm not seeing why not to support this.\n\nThanks,\nJonathan\n"},{"id":"341046","messageId":"20180305184321.GC72475@google.com","threadId":"47524","inReplyTo":"20180303043338.GC27689@sigill.intra.peff.net","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-05T18:43:21Z","receivedAt":"2018-03-05T18:43:29Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/02, Jeff King wrote:\n> On Fri, Feb 23, 2018 at 01:45:57PM -0800, Brandon Williams wrote:\n> \n> > I think this is the price of extending the protocol in a backward\n> > compatible way. If we don't want to be backwards compatible (allowing\n> > for graceful fallback to v1) then we could design this differently.\n> > Even so we're not completely out of luck just yet.\n> > \n> > Back when I introduced the GIT_PROTOCOL side-channel I was able to\n> > demonstrate that arbitrary data could be sent to the server and it would\n> > only respect the stuff it knows about. This means that we can do a\n> > follow up to v2 at some point to introduce an optimization where we can\n> > stuff a request into GIT_PROTOCOL and short-circuit the first round-trip\n> > if the server supports it.\n> \n> If that's our end-game, it does make me wonder if we'd be happier just\n> jumping to that at first. Before you started the v2 protocol work, I had\n> a rough patch series passing what I called \"early capabilities\". The\n> idea was to let the client speak a few optional capabilities before the\n> ref advertisement, and be ready for the server to ignore them\n> completely. That doesn't clean up all the warts with the v0 protocol,\n> but it handles the major one (allowing more efficient ref\n> advertisements).\n\nI didn't really want to get to that just yet, simply because I want to\ntry and keep the scope of this smaller while still being able to fix\nmost of the issues we have with v0.\n\n> I dunno. There's a lot more going on here in v2 and I'm not sure I've\n> fully digested it.\n\nI tried to keep it similar enough to v0 such that it wouldn't be that\nbig of a leap (small steps). For example negotiation is really done the\nsame as it is in v0 during fetch (a next step would be to actually\nimprove that). We can definitely talk about all this in more detail\nlater this week too.\n\n> \n> > The great thing about this is that from the POV of the git-client, it\n> > doesn't care if its speaking using the git://, ssh://, file://, or\n> > http:// transport; it's all the same protocol. In my next re-roll I'll\n> > even drop the \"# service\" bit from the http server response and then the\n> > responses will truly be identical in all cases.\n> \n> This part has me a little confused still. The big difference between\n> http and the other protocols is that the other ones are full-duplex, and\n> http is a series of stateless request/response pairs.\n> \n> Are the other protocols becoming stateless request/response pairs, too?\n> Or will they be \"the same protocol\" only in the sense of using the same\n> transport?\n> \n> (There are a lot of reasons not to like the stateless pair thing; it has\n> some horrid corner cases during want/have negotiation).\n\nJunio made a comment on the Spec in the most recent version of the\nseries about how I state that v2 is stateless and \"MUST NOT\" rely on\nstate being stored on the server side. In reality I think this needs to\nbe tweaked a bit because when you do have a full-duplex connection you\nmay probably want to use that to reduce the amount of data that you send\nin some cases.\n\nIn the current protocol http has a lot of additional stuff that's had to\nbe done to it to get it to work with a protocol that was designed to be\nstateful first. What I want is for the protocol to be designed\nstateless first so that http functions essentially the same as ssh or\nfile or git transports and we don't have to do any hackery to get it to\nwork. This also makes it very simple to implement a new feature in the\nprotocol because you only need to think about implementing it once\ninstead of twice like you kind of have to do with v0. So in the most\nrecent series everything is a chain of request/response pairs even in\nthe non-http cases.\n\nIn a previous version of the series I had each command being able to\nlast any number of rounds and having a 'stateless' capability indicating\nif the command needed to be run stateless. I didn't think that was a\ngood design because by default you are still designing the stateful\nthing first and the http (stateless) case can be an afterthought. So\ninstead maybe we'll need commands which can benefit from state to have a\n'stateful' feature that can be advertised when a full-duplex connection\nis possible. This still gives you the opportunity to not advertise that\nand have the same behavior over ssh as http. I actually remember\nhearing someone talk about how they would like to allow for ssh\nconnections to their server and just have it be a proxy for http and\nthis would enable that.\n\n-- \nBrandon Williams\n"},{"id":"341048","messageId":"20180305184848.GA162172@google.com","threadId":"47524","inReplyTo":"20180303042558.GB27689@sigill.intra.peff.net","subject":"Re: [PATCH v3 11/35] test-pkt-line: introduce a packet-line test helper","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-05T18:48:48Z","receivedAt":"2018-03-05T18:48:56Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/02, Jeff King wrote:\n> On Fri, Feb 23, 2018 at 01:22:31PM -0800, Brandon Williams wrote:\n> \n> > On 02/22, Stefan Beller wrote:\n> > > On Tue, Feb 6, 2018 at 5:12 PM, Brandon Williams wrote:\n> > > \n> > > > +static void pack_line(const char *line)\n> > > > +{\n> > > > + if (!strcmp(line, \"0000\") || !strcmp(line, \"0000\\n\"))\n> > > \n> > > From our in-office discussion:\n> > > v1/v0 packs pktlines twice in http, which is not possible to\n> > > construct using this test helper when using the same string\n> > > for the packed and unpacked representation of flush and delim packets,\n> > > i.e. test-pkt-line --pack $(test-pkt-line --pack 0000) would produce\n> > > '0000' instead of '00090000\\n'.\n> > > To fix it we'd have to replace the unpacked versions of these pkts to\n> > > something else such as \"FLUSH\" \"DELIM\".\n> > > \n> > > However as we do not anticipate the test helper to be used in further\n> > > tests for v0, this ought to be no big issue.\n> > > Maybe someone else cares though?\n> > \n> > I'm going to punt and say, if someone cares enough they can update this\n> > test-helper when they want to use it for v1/v0 stuff.\n> \n> I recently add packetize and depacketize helpers for testing v0 streams;\n> see 4414a15002 (t/lib-git-daemon: add network-protocol helpers,\n> 2018-01-24). Is it worth folding these together?\n\nI didn't know something like that existed! (of course if it was just\nadded this year then it didn't exist when I started working on this\nstuff). Yeah its probably a good idea to fold these together, I can\ntake a look at how your packetize and depacketize helpers work and add\nthe small amount of functionality that I'd need to replace the helper I\nmade.\n\n> \n> -Peff\n\n-- \nBrandon Williams\n"},{"id":"341054","messageId":"20180305193506.GB162172@google.com","threadId":"47524","inReplyTo":"nycvar.QRO.7.76.6.1803022106030.56@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz","subject":"Re: [PATCH v4 34/35] remote-curl: implement stateless-connect command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-05T19:35:06Z","receivedAt":"2018-03-05T19:35:14Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/02, Johannes Schindelin wrote:\n> Hi Brandon,\n> \n> On Wed, 28 Feb 2018, Brandon Williams wrote:\n> \n> > diff --git a/remote-curl.c b/remote-curl.c\n> > index 66a53f74b..3f882d766 100644\n> > --- a/remote-curl.c\n> > +++ b/remote-curl.c\n> > @@ -188,7 +188,12 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n> > [...]\n> > +static size_t proxy_in(char *buffer, size_t eltsize,\n> > +\t\t size_t nmemb, void *userdata)\n> > +{\n> > +\tsize_t max;\n> > +\tstruct proxy_state *p = userdata;\n> > +\tsize_t avail = p->request_buffer.len - p->pos;\n> > +\n> > +\n> > +\tif (eltsize != 1)\n> > +\t\tBUG(\"curl read callback called with size = %zu != 1\", eltsize);\n> \n> The format specified %z is not actually portable. Please use PRIuMAX and\n> cast to (uintmax_t) instead.\n> \n> This breaks the Windows build of `pu` (before that, there was still a test\n> failure that I did not have the time to chase down).\n\nOh sorry, Looks like Junio put a patch ontop in pu to fix this. I'll\nsquash that fix into this patch.\n\nThanks for catching this.\n\n> \n> Ciao,\n> Dscho\n\n-- \nBrandon Williams\n"},{"id":"341060","messageId":"20180305202813.GA5953@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180305182155.GB72475@google.com","subject":"Re: [PATCH v4 13/35] ls-refs: introduce ls-refs server command","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-05T20:28:13Z","receivedAt":"2018-03-05T20:28:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 05, 2018 at 10:21:55AM -0800, Brandon Williams wrote:\n\n> > Hmm, so this would accept stuff like \"refs/heads/*/foo\" but quietly\n> > ignore the \"/foo\" part.\n> \n> Yeah that's true...this should probably not do that. Since\n> \"refs/heads/*/foo\" violates what the spec is, really this should error\n> out as an invalid pattern.\n\nYeah, that would be better, I think.\n\n> > It also accepts \"refs/h*\" to get \"refs/heads\" and \"refs/hello\". I think\n> > it's worth going for the most-restrictive thing to start with, since\n> > that enables a lot more server operations without worrying about\n> > breaking compatibility.\n> \n> And just to clarify what do you see as being the most-restrictive case\n> of patterns that would should use?\n\nI mean only accepting \"*\" at a \"/\" boundary (or just allowing a trailing\nslash to imply recursion, like \"refs/heads/\", or even just always\nassuming recursion to allow \"refs/heads\").\n\n-Peff\n"},{"id":"341061","messageId":"20180305203802.GB5953@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180305182914.GD28067@aiede.svl.corp.google.com","subject":"Re: [PATCH v4 13/35] ls-refs: introduce ls-refs server command","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-05T20:38:02Z","receivedAt":"2018-03-05T20:38:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 05, 2018 at 10:29:14AM -0800, Jonathan Nieder wrote:\n\n> >> It also accepts \"refs/h*\" to get \"refs/heads\" and \"refs/hello\". I think\n> >> it's worth going for the most-restrictive thing to start with, since\n> >> that enables a lot more server operations without worrying about\n> >> breaking compatibility.\n> >\n> > And just to clarify what do you see as being the most-restrictive case\n> > of patterns that would should use?\n> \n> Peff, can you say a little more about the downsides of accepting\n> refs/h*?\n> \n> IIRC the \"git push\" command already accepts such refspecs, so there's a\n> benefit to accepting them. Reftable and packed-refs support such\n> queries about as efficiently as refs/heads/*. For loose refs, readdir\n> doesn't provide a way to restrict which files you look at, but loose\n> refs are always slow anyway. :)\n> \n> In other words, I see real benefits and I don't see much in the way of\n> costs, so I'm not seeing why not to support this.\n\n\"git for-each-ref\" only handles \"/\" boundaries. I think we used to have\nsimilar problems with the internal for_each_ref(), but I just checked\nand I think it's more flexible these days. One could imagine a more\ntrie-like storage, though I agree that is stretching it with a\nhypothetical.\n\nMostly my point was that I don't see any big upside, and the choice\nseemed rather arbitrary. And as it is generally easier to loosen the\npatterns later than tighten them, it makes sense to go with the tightest\noption at first unless there is a compelling reason not to.\n\n-Peff\n"},{"id":"341062","messageId":"20180305205254.GC5953@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180305184321.GC72475@google.com","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-05T20:52:55Z","receivedAt":"2018-03-05T20:53:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 05, 2018 at 10:43:21AM -0800, Brandon Williams wrote:\n\n> In the current protocol http has a lot of additional stuff that's had to\n> be done to it to get it to work with a protocol that was designed to be\n> stateful first. What I want is for the protocol to be designed\n> stateless first so that http functions essentially the same as ssh or\n> file or git transports and we don't have to do any hackery to get it to\n> work. This also makes it very simple to implement a new feature in the\n> protocol because you only need to think about implementing it once\n> instead of twice like you kind of have to do with v0. So in the most\n> recent series everything is a chain of request/response pairs even in\n> the non-http cases.\n\nI agree that would be a lot more pleasant for adding protocol features.\nBut I just worry that the stateful protocols get a lot less efficient.\nI'm having trouble coming up with an easy reproduction, but my\nrecollection is that http has some nasty corner cases, because each\nround of \"have\" lines sent to the server has to summarize the previous\nconversation. So you can get a case where the client's requests keep\ngetting bigger and bigger during the negotiation (and eventually getting\nlarge enough to cause problems).\n\nIf anything, I wish we could push the http protocol in a more stateful\ndirection with something like websockets. But I suspect that's an\nunrealistic dream, just because not everybody's http setup (proxies,\netc) will be able to handle that.\n\n-Peff\n"},{"id":"341069","messageId":"20180305213649.GF28067@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180305205254.GC5953@sigill.intra.peff.net","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-03-05T21:36:49Z","receivedAt":"2018-03-05T21:37:00Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff King wrote:\n\n> I agree that would be a lot more pleasant for adding protocol features.\n> But I just worry that the stateful protocols get a lot less efficient.\n> I'm having trouble coming up with an easy reproduction, but my\n> recollection is that http has some nasty corner cases, because each\n> round of \"have\" lines sent to the server has to summarize the previous\n> conversation. So you can get a case where the client's requests keep\n> getting bigger and bigger during the negotiation (and eventually getting\n> large enough to cause problems).\n\nThat's not so much a corner case as just how negotiation works over\nhttp.\n\nWe want to do better (e.g. see [1]) but that's a bigger change than\nthe initial protocol v2.\n\nAs Brandon explained it to me, we really do want to use stateless-rpc\nsemantics by default, since that's just better for maintainability.\nInstead of having two protocols, one that is sane and one that\nstruggles to hoist that into stateless-rpc, there would be one\nstateless baseline plus capabilities to make use of state.\n\nFor example, it would be nice to have a capability to remember\nnegotiation state between rounds, to get around exactly the problem\nyou're describing when using a stateful protocol. Stateless backends\nwould just not advertise such a capability. But doing that without [1]\nstill sort of feels like a cop-out. If we can get a reasonable\nbaseline using ideas like [1] and then have a capability to keep\nserver-side state as icing on the cake instead of having a negotiation\nprocess that only really makes sense when you have server-side state,\nthen that would be even better.\n\n> If anything, I wish we could push the http protocol in a more stateful\n> direction with something like websockets. But I suspect that's an\n> unrealistic dream, just because not everybody's http setup (proxies,\n> etc) will be able to handle that.\n\nAgreed. I think we have to continue to deal with stateless-rpc\nsemantics, at least for the near future.\n\nJonathan\n\n[1] https://public-inbox.org/git/20180227054638.GB65699@aiede.svl.corp.google.com/\n"},{"id":"341098","messageId":"20180306062901.GA3522@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180305213649.GF28067@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-06T06:29:02Z","receivedAt":"2018-03-06T06:29:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 05, 2018 at 01:36:49PM -0800, Jonathan Nieder wrote:\n\n> > I agree that would be a lot more pleasant for adding protocol features.\n> > But I just worry that the stateful protocols get a lot less efficient.\n> > I'm having trouble coming up with an easy reproduction, but my\n> > recollection is that http has some nasty corner cases, because each\n> > round of \"have\" lines sent to the server has to summarize the previous\n> > conversation. So you can get a case where the client's requests keep\n> > getting bigger and bigger during the negotiation (and eventually getting\n> > large enough to cause problems).\n> \n> That's not so much a corner case as just how negotiation works over\n> http.\n\nSure. What I meant more was \"there are corner cases where it gets out of\ncontrol and doesn't work\".\n\nI have had to give the advice in the past \"if your fetch over http\ndoesn't work, try it over ssh\". If we change the ssh protocol to be\nstateless, too, then that closes that escape hatch.\n\nI haven't had to give that advice for a while, though. Maybe tweaks to\nthe parameters or just larger buffers have made the problem go away over\nthe years?\n\n> We want to do better (e.g. see [1]) but that's a bigger change than\n> the initial protocol v2.\n> \n> As Brandon explained it to me, we really do want to use stateless-rpc\n> semantics by default, since that's just better for maintainability.\n> Instead of having two protocols, one that is sane and one that\n> struggles to hoist that into stateless-rpc, there would be one\n> stateless baseline plus capabilities to make use of state.\n\nYes, I think that would be a nice end-game. It just wasn't clear to me\nwhere we'd be in the interim.\n\n-Peff\n"},{"id":"341495","messageId":"20180312215635.GA61720@google.com","threadId":"47524","inReplyTo":"xmqqk1uvfro3.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 02/35] pkt-line: allow peeking a packet line without consuming it","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-12T21:56:35Z","receivedAt":"2018-03-12T21:56:43Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/01, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > +enum packet_read_status packet_reader_read(struct packet_reader *reader)\n> > +{\n> > +\tif (reader->line_peeked) {\n> > +\t\treader->line_peeked = 0;\n> > +\t\treturn reader->status;\n> > +\t}\n> > +\n> > +\treader->status = packet_read_with_status(reader->fd,\n> > +\t\t\t\t\t\t &reader->src_buffer,\n> > +\t\t\t\t\t\t &reader->src_len,\n> > +\t\t\t\t\t\t reader->buffer,\n> > +\t\t\t\t\t\t reader->buffer_size,\n> > +\t\t\t\t\t\t &reader->pktlen,\n> > +\t\t\t\t\t\t reader->options);\n> > +\n> > +\tswitch (reader->status) {\n> > +\tcase PACKET_READ_EOF:\n> > +\t\treader->pktlen = -1;\n> > +\t\treader->line = NULL;\n> > +\t\tbreak;\n> > +\tcase PACKET_READ_NORMAL:\n> > +\t\treader->line = reader->buffer;\n> > +\t\tbreak;\n> > +\tcase PACKET_READ_FLUSH:\n> > +\t\treader->pktlen = 0;\n> > +\t\treader->line = NULL;\n> > +\t\tbreak;\n> > +\t}\n> > +\n> > +\treturn reader->status;\n> > +}\n> \n> With the way _peek() interface interacts with the reader instance\n> (which by the way I find is well designed), it is understandable\n> that we want almost everything available in reader's fields, but\n> having to manually clear pktlen field upon non NORMAL status feels\n> a bit strange. \n> \n> Perhaps that is because the underlying packet_read_with_status()\n> does not set *pktlen in these cases? Shouldn't it be doing that so\n> the caller does not have to?\n\nThat's true, I'll fix that.\n\n\n-- \nBrandon Williams\n"},{"id":"341497","messageId":"20180312220808.GB61720@google.com","threadId":"47524","inReplyTo":"xmqqinafe6ih.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 12/35] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-12T22:08:08Z","receivedAt":"2018-03-12T22:08:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/01, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > Documentation/technical/protocol-v2.txt | 171 ++++++++++++++++\n> \n> Unlike other things in Documentation/technical/, this is not listed\n> on TECH_DOCS list in Documentation/Makefile. Shouldn't it be?\n\nYes it should, I'll fix that.\n\n-- \nBrandon Williams\n"},{"id":"341498","messageId":"20180312221855.GC61720@google.com","threadId":"47524","inReplyTo":"xmqq4llyazma.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 18/35] fetch: pass ref patterns when fetching","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-12T22:18:55Z","receivedAt":"2018-03-12T22:19:02Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/02, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > diff --git a/builtin/fetch.c b/builtin/fetch.c\n> > index 850382f55..695fafe06 100644\n> > --- a/builtin/fetch.c\n> > +++ b/builtin/fetch.c\n> > @@ -332,11 +332,25 @@ static struct ref *get_ref_map(struct transport *transport,\n> > \tstruct ref *rm;\n> > \tstruct ref *ref_map = NULL;\n> > \tstruct ref **tail = &ref_map;\n> > +\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n> > \n> > \t/* opportunistically-updated references: */\n> > \tstruct ref *orefs = NULL, **oref_tail = &orefs;\n> > \n> > -\tconst struct ref *remote_refs = transport_get_remote_refs(transport, NULL);\n> > +\tconst struct ref *remote_refs;\n> > +\n> > +\tfor (i = 0; i < refspec_count; i++) {\n> > +\t\tif (!refspecs[i].exact_sha1) {\n> > +\t\t\tif (refspecs[i].pattern)\n> > +\t\t\t\targv_array_push(&ref_patterns, refspecs[i].src);\n> > +\t\t\telse\n> > +\t\t\t\texpand_ref_pattern(&ref_patterns, refspecs[i].src);\n> > +\t\t}\n> > +\t}\n> > +\n> > +\tremote_refs = transport_get_remote_refs(transport, &ref_patterns);\n> > +\n> > +\targv_array_clear(&ref_patterns);\n> \n> Is the idea here, which is shared with 17/35 about ls-remote, that\n> we used to grab literally everything they have in remote_refs, but\n> we have code in place to filter that set using refspecs given in the\n> remote.*.fetch configuration, so it is OK as long as we grab everything\n> that would match the remote.*.fetch pattern? That is, grabbing too\n> much is acceptable, but if we populated ref_patterns[] with too few\n> patterns and fail to ask refs that would match our refspec it would\n> be a bug?\n\nYes that's the idea. Right now we're in the state where we ask for\neverything (since there is no server side filtering) and the client just\ndoes its own filtering after the fact using the refspec. So if we end\nup not sending enough ref patterns to match what the refspec is, it\nwould be a bug.\n\n> \n> The reason behind this question is that I am wondering if/how we can\n> take advantage of this remote-side pre-filtering while doing \"fetch\n> --prune\".\n\nHmm maybe, assuming prune then means \"get rid of all remote-tracking\nbranches that don't match the user provided refspec\"\n\n> \n> Thanks.\n> \n> > \n> > \tif (refspec_count) {\n> > \t\tstruct refspec *fetch_refspec;\n\n-- \nBrandon Williams\n"},{"id":"341500","messageId":"20180312222007.GD61720@google.com","threadId":"47524","inReplyTo":"xmqqzi3q9ku6.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 19/35] push: pass ref patterns when pushing","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-12T22:20:07Z","receivedAt":"2018-03-12T22:20:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/02, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > Construct a list of ref patterns to be passed to 'get_refs_list()' from\n> > the refspec to be used during the push. This list of ref patterns will\n> > be used to allow the server to filter the ref advertisement when\n> > communicating using protocol v2.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > transport.c | 26 +++++++++++++++++++++++++-\n> > 1 file changed, 25 insertions(+), 1 deletion(-)\n> \n> When you are pushing 'master', we no longer hear what the other end\n> has at 'next', with this change, no?\n> \n> In a project whose 'master' is extended primarily by merging topics\n> that have been cooking in 'next', old way of pushing would only have\n> transferred the merge commits and resulting trees but not bulk of\n> the blob data because they are all available on 'next', would it\n> make the object transfer far less efficient, I wonder?\n> \n> I guess it is OK only because the push side of the current protocol\n> does not do common ancestor discovery exchange ;-)\n\nYep, though we've been throwing around ideas of adding that in push v2\nafter we figure out a good way to improve negotiation with fetch.\n\n> \n> >\n> > diff --git a/transport.c b/transport.c\n> > index dfc603b36..bf7ba6879 100644\n> > --- a/transport.c\n> > +++ b/transport.c\n> > @@ -1026,11 +1026,35 @@ int transport_push(struct transport *transport,\n> > \t\tint porcelain = flags & TRANSPORT_PUSH_PORCELAIN;\n> > \t\tint pretend = flags & TRANSPORT_PUSH_DRY_RUN;\n> > \t\tint push_ret, ret, err;\n> > +\t\tstruct refspec *tmp_rs;\n> > +\t\tstruct argv_array ref_patterns = ARGV_ARRAY_INIT;\n> > +\t\tint i;\n> > \n> > \t\tif (check_push_refs(local_refs, refspec_nr, refspec) < 0)\n> > \t\t\treturn -1;\n> > \n> > -\t\tremote_refs = transport->vtable->get_refs_list(transport, 1, NULL);\n> > +\t\ttmp_rs = parse_push_refspec(refspec_nr, refspec);\n> > +\t\tfor (i = 0; i < refspec_nr; i++) {\n> > +\t\t\tconst char *pattern = NULL;\n> > +\n> > +\t\t\tif (tmp_rs[i].dst)\n> > +\t\t\t\tpattern = tmp_rs[i].dst;\n> > +\t\t\telse if (tmp_rs[i].src && !tmp_rs[i].exact_sha1)\n> > +\t\t\t\tpattern = tmp_rs[i].src;\n> > +\n> > +\t\t\tif (pattern) {\n> > +\t\t\t\tif (tmp_rs[i].pattern)\n> > +\t\t\t\t\targv_array_push(&ref_patterns, pattern);\n> > +\t\t\t\telse\n> > +\t\t\t\t\texpand_ref_pattern(&ref_patterns, pattern);\n> > +\t\t\t}\n> > +\t\t}\n> > +\n> > +\t\tremote_refs = transport->vtable->get_refs_list(transport, 1,\n> > +\t\t\t\t\t\t\t &ref_patterns);\n> > +\n> > +\t\targv_array_clear(&ref_patterns);\n> > +\t\tfree_refspec(refspec_nr, tmp_rs);\n> > \n> > \t\tif (flags & TRANSPORT_PUSH_ALL)\n> > \t\t\tmatch_flags |= MATCH_REFS_ALL;\n\n-- \nBrandon Williams\n"},{"id":"341501","messageId":"20180312222443.GE61720@google.com","threadId":"47524","inReplyTo":"xmqq371jfpzj.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 05/35] upload-pack: factor out processing lines","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-12T22:24:43Z","receivedAt":"2018-03-12T22:24:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/01, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > Factor out the logic for processing shallow, deepen, deepen_since, and\n> > deepen_not lines into their own functions to simplify the\n> > 'receive_needs()' function in addition to making it easier to reuse some\n> > of this logic when implementing protocol_v2.\n> \n> These little functions that still require their incoming data to\n> begin with fixed prefixes feels a bit strange way to refactor the\n> logic for later reuse (when I imagine \"reuse\", the first use case\n> that comes to my mind is \"this data source our new code reads from\n> gives the same data as the old 'shallow' packet used to give, but in\n> a different syntax\"---so I'd restructure the code in such a way that\n> the caller figures out the syntax part and the called helper just\n> groks the \"information contents\" unwrapped from the surface syntax;\n> the syntax may be different in the new codepath but once unwrapped,\n> the \"information contents\" to be processed would not be different\n> hence we can reuse the helper).\n> \n> IOW, I would have expected the caller to be not like this:\n> \n> > -\t\tif (skip_prefix(line, \"shallow \", &arg)) {\n> > -\t\t\tstruct object_id oid;\n> > -\t\t\tstruct object *object;\n> > -\t\t\tif (get_oid_hex(arg, &oid))\n> > -\t\t\t\tdie(\"invalid shallow line: %s\", line);\n> > -\t\t\tobject = parse_object(&oid);\n> > -\t\t\tif (!object)\n> > -\t\t\t\tcontinue;\n> > -\t\t\tif (object->type != OBJ_COMMIT)\n> > -\t\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n> > -\t\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n> > -\t\t\t\tobject->flags |= CLIENT_SHALLOW;\n> > -\t\t\t\tadd_object_array(object, NULL, &shallows);\n> > -\t\t\t}\n> > +\t\tif (process_shallow(line, &shallows))\n> > \t\t\tcontinue;\n> > +\t\tif (process_deepen(line, &depth))\n> > \t\t\tcontinue;\n> \t\t...\n> \n> but more like\n> \n> \t\tif (skip_prefix(line, \"shallow \", &arg) {\n> \t\t\tprocess_shallow(arg, &shallows);\n> \t\t\tcontinue;\n> \t\t}\n> \t\tif (skip_prefix(line, \"deepen \", &arg) {\n> \t\t\tprocess_deepen(arg, &depth);\n> \t\t\tcontinue;\n> \t\t}\n> \t\t...\n> \n> I need to defer the final judgment until I see how they are used,\n> though. It's not too big a deal either way---it just felt \"not\n> quite right\" to me.\n\nThis is actually a really good point (and maybe the same point stefan\nwas trying to make on an old revision of this series). I think it makes\nmuch more sense to refactor the code to have a structure like you've\noutlined. I'll fix this for the next version.\n\n-- \nBrandon Williams\n"},{"id":"341502","messageId":"20180312223946.GF61720@google.com","threadId":"47524","inReplyTo":"20180312222443.GE61720@google.com","subject":"Re: [PATCH v4 05/35] upload-pack: factor out processing lines","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-12T22:39:46Z","receivedAt":"2018-03-12T22:39:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/12, Brandon Williams wrote:\n> On 03/01, Junio C Hamano wrote:\n> > Brandon Williams writes:\n> > \n> > > Factor out the logic for processing shallow, deepen, deepen_since, and\n> > > deepen_not lines into their own functions to simplify the\n> > > 'receive_needs()' function in addition to making it easier to reuse some\n> > > of this logic when implementing protocol_v2.\n> > \n> > These little functions that still require their incoming data to\n> > begin with fixed prefixes feels a bit strange way to refactor the\n> > logic for later reuse (when I imagine \"reuse\", the first use case\n> > that comes to my mind is \"this data source our new code reads from\n> > gives the same data as the old 'shallow' packet used to give, but in\n> > a different syntax\"---so I'd restructure the code in such a way that\n> > the caller figures out the syntax part and the called helper just\n> > groks the \"information contents\" unwrapped from the surface syntax;\n> > the syntax may be different in the new codepath but once unwrapped,\n> > the \"information contents\" to be processed would not be different\n> > hence we can reuse the helper).\n> > \n> > IOW, I would have expected the caller to be not like this:\n> > \n> > > -\t\tif (skip_prefix(line, \"shallow \", &arg)) {\n> > > -\t\t\tstruct object_id oid;\n> > > -\t\t\tstruct object *object;\n> > > -\t\t\tif (get_oid_hex(arg, &oid))\n> > > -\t\t\t\tdie(\"invalid shallow line: %s\", line);\n> > > -\t\t\tobject = parse_object(&oid);\n> > > -\t\t\tif (!object)\n> > > -\t\t\t\tcontinue;\n> > > -\t\t\tif (object->type != OBJ_COMMIT)\n> > > -\t\t\t\tdie(\"invalid shallow object %s\", oid_to_hex(&oid));\n> > > -\t\t\tif (!(object->flags & CLIENT_SHALLOW)) {\n> > > -\t\t\t\tobject->flags |= CLIENT_SHALLOW;\n> > > -\t\t\t\tadd_object_array(object, NULL, &shallows);\n> > > -\t\t\t}\n> > > +\t\tif (process_shallow(line, &shallows))\n> > > \t\t\tcontinue;\n> > > +\t\tif (process_deepen(line, &depth))\n> > > \t\t\tcontinue;\n> > \t\t...\n> > \n> > but more like\n> > \n> > \t\tif (skip_prefix(line, \"shallow \", &arg) {\n> > \t\t\tprocess_shallow(arg, &shallows);\n> > \t\t\tcontinue;\n> > \t\t}\n> > \t\tif (skip_prefix(line, \"deepen \", &arg) {\n> > \t\t\tprocess_deepen(arg, &depth);\n> > \t\t\tcontinue;\n> > \t\t}\n> > \t\t...\n> > \n> > I need to defer the final judgment until I see how they are used,\n> > though. It's not too big a deal either way---it just felt \"not\n> > quite right\" to me.\n> \n> This is actually a really good point (and maybe the same point stefan\n> was trying to make on an old revision of this series). I think it makes\n> much more sense to refactor the code to have a structure like you've\n> outlined. I'll fix this for the next version.\n\nAnd then I started writing the code and now I don't know which I\nprefer. The issue is that its for processing a line which has some well\ndefined structure and moving the check for \"shallow \" away from the rest\nof the code which does the processing makes it a little less clear how\nthat shallow line is to be defined.\n\n-- \nBrandon Williams\n"},{"id":"341504","messageId":"20180312224355.GC58506@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180222214457.GB30638@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-03-12T22:43:55Z","receivedAt":"2018-03-12T22:44:03Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff King wrote:\n> On Thu, Feb 22, 2018 at 01:26:34PM -0800, Jonathan Nieder wrote:\n\n>> Keep in mind that git upload-archive (a read-only command, just like\n>> git upload-pack) also already has the same issues.\n>\n> Yuck. I don't think we've ever made a historical promise about that. But\n> then, I don't think the promise about upload-pack has ever really been\n> documented, except in mailing list discussions.\n\nSorry to revive this old side-thread. Good news: for a dashed command\nlike git-upload-archive, the pager selection code only runs for\ncommands with RUN_SETUP or RUN_SETUP_GENTLY:\n\n\tif (use_pager == -1 && p->option & (RUN_SETUP | RUN_SETUP_GENTLY) &&\n\t !(p->option & DELAY_PAGER_CONFIG))\n\t\tuse_pager = check_pager_config(p->cmd);\n\nNone of upload-pack, receive-pack,git-serve, or upload-archive set\nthose flags, so we (narrowly) escape trouble here.\n\nLater today I should be able to send a cleanup to make the behavior\nmore obvious.\n\nThanks again,\nJonathan\n"},{"id":"341507","messageId":"20180312232844.GB1968@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180312224355.GC58506@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-12T23:28:44Z","receivedAt":"2018-03-12T23:28:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 12, 2018 at 03:43:55PM -0700, Jonathan Nieder wrote:\n\n> Hi,\n> \n> Jeff King wrote:\n> > On Thu, Feb 22, 2018 at 01:26:34PM -0800, Jonathan Nieder wrote:\n> \n> >> Keep in mind that git upload-archive (a read-only command, just like\n> >> git upload-pack) also already has the same issues.\n> >\n> > Yuck. I don't think we've ever made a historical promise about that. But\n> > then, I don't think the promise about upload-pack has ever really been\n> > documented, except in mailing list discussions.\n> \n> Sorry to revive this old side-thread. Good news: for a dashed command\n> like git-upload-archive, the pager selection code only runs for\n> commands with RUN_SETUP or RUN_SETUP_GENTLY:\n> \n> \tif (use_pager == -1 && p->option & (RUN_SETUP | RUN_SETUP_GENTLY) &&\n> \t !(p->option & DELAY_PAGER_CONFIG))\n> \t\tuse_pager = check_pager_config(p->cmd);\n> \n> None of upload-pack, receive-pack,git-serve, or upload-archive set\n> those flags, so we (narrowly) escape trouble here.\n\nRight, I saw that earlier. But I actually think that is stale from the\ndays when it wasn't safe to call check_pager_config() too early. So I\ncould very well see somebody removing it and causing a spooky\nvulnerability at a distance.\n\n> Later today I should be able to send a cleanup to make the behavior\n> more obvious.\n\nThanks. I'm still on the fence over the whole builtin concept, but\ncertainly a \"don't ever turn on a pager\" flag seems like a reasonable\nthing to have.\n\nAn alternative approach is some kind of global for \"don't trust the\nlocal repo\" flag. That could be respected from very low-level code\n(e.g., where we read and/or respect the pager command, but also in other\nplaces like hooks, other config that runs arbitrary commands, etc). And\nthen upload-pack would set that to \"do not trust\", and other programs\nwould default to \"trust\".\n\nWe could even give it an environment variable, which would allow\nsomething like:\n\n tar xf maybe-evil.git.tar\n cd maybe-evil\n export GIT_TRUST_REPO=false\n git log\n\nwithout worrying about pager.log config, etc. My two concerns with this\napproach would be:\n\n 1. We have to manually annotate any \"dangerous\" code to act more\n safely when it sees the flag. Which means it's highly likely to\n a spot, or to add a new feature which doesn't respect it. And\n suddenly that's a security hole. So I'm concerned it may create a\n false sense of security and actually make things worse.\n\n 2. As a global, I'm not sure how it would interact with multi-repo\n processes like submodules. In theory it ought to go into the\n repository struct, but it would often need to be set globally\n before we've even discovered the repo.\n\n That might be fine, though. It's really more about context than\n about a specific repo (so you may say \"don't trust this repo\", and\n that extends to any submodules you happen to access, too).\n\nI dunno. I think (2) is probably OK, but (1) really gives me pause.\n\n-Peff\n"},{"id":"341509","messageId":"20180312233747.GB235366@aiede.svl.corp.google.com","threadId":"47524","inReplyTo":"20180312232844.GB1968@sigill.intra.peff.net","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-03-12T23:37:47Z","receivedAt":"2018-03-12T23:37:55Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff King wrote:\n\n> We could even give it an environment variable, which would allow\n> something like:\n>\n> tar xf maybe-evil.git.tar\n> cd maybe-evil\n> export GIT_TRUST_REPO=false\n> git log\n\nInteresting idea. Putting it in an envvar means it gets inherited by\nchild processes, which if I understand you correctly is a good thing.\n\n[...]\n> 1. We have to manually annotate any \"dangerous\" code to act more\n> safely when it sees the flag. Which means it's highly likely to\n> a spot, or to add a new feature which doesn't respect it. And\n> suddenly that's a security hole. So I'm concerned it may create a\n> false sense of security and actually make things worse.\n\nAs an internal implementation detail, this is so obviously fragile\nthat it wouldn't give me any feeling of security. ;-) So it should be\nstrictly an improvement.\n\nAs a public-facing feature, I suspect it's a bad idea for exactly that\nreason.\n\nFWIW for pager specifically I am going for a whitelisting approach:\nnew commands would have to explicitly set ALLOW_PAGER if they want to\nrespect pager config. That doesn't guarantee people think about it\nagain as things evolve but it should at least help with getting the\nright setting for new plumbing.\n\nThanks,\nJonathan\n"},{"id":"341511","messageId":"20180312234651.GA2137@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180306062901.GA3522@sigill.intra.peff.net","subject":"Re: [PATCH v3 12/35] serve: introduce git-serve","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-12T23:46:51Z","receivedAt":"2018-03-12T23:46:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 06, 2018 at 07:29:02AM +0100, Jeff King wrote:\n\n> > We want to do better (e.g. see [1]) but that's a bigger change than\n> > the initial protocol v2.\n> > \n> > As Brandon explained it to me, we really do want to use stateless-rpc\n> > semantics by default, since that's just better for maintainability.\n> > Instead of having two protocols, one that is sane and one that\n> > struggles to hoist that into stateless-rpc, there would be one\n> > stateless baseline plus capabilities to make use of state.\n> \n> Yes, I think that would be a nice end-game. It just wasn't clear to me\n> where we'd be in the interim.\n\nAfter some more thinking about this, and a little chatting with Brandon\nat the contrib summit, I'm willing to soften my position on this.\n\nBasically I was concerned about this as a regression where git-over-ssh\nwould stop working in a few corner cases. And it would cease to be\navailable as an escape hatch for those cases where http wouldn't work.\n\nBut we may be OK in this \"interim\" period (before unified\nstateful-negotiation bits are added back) because v2 would not yet be\nthe default. So the ssh cases can't regress without flipping the v2\nswitch manually, and any escape hatch would continue to work by flipping\nback to v1 anyway.\n\nSo it's probably OK to continue experimenting in this direction and see\nhow often it's a problem in practice.\n\n-Peff\n"},{"id":"341512","messageId":"20180312235200.GF1968@sigill.intra.peff.net","threadId":"47524","inReplyTo":"20180312233747.GB235366@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 04/35] upload-pack: convert to a builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-03-12T23:52:00Z","receivedAt":"2018-03-12T23:52:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 12, 2018 at 04:37:47PM -0700, Jonathan Nieder wrote:\n\n> Jeff King wrote:\n> \n> > We could even give it an environment variable, which would allow\n> > something like:\n> >\n> > tar xf maybe-evil.git.tar\n> > cd maybe-evil\n> > export GIT_TRUST_REPO=false\n> > git log\n> [...]\n> As an internal implementation detail, this is so obviously fragile\n> that it wouldn't give me any feeling of security. ;-) So it should be\n> strictly an improvement.\n> \n> As a public-facing feature, I suspect it's a bad idea for exactly that\n> reason.\n\nSo that pretty much kills off the GIT_TRUST_REPO idea, I guess.\n\n> FWIW for pager specifically I am going for a whitelisting approach:\n> new commands would have to explicitly set ALLOW_PAGER if they want to\n> respect pager config. That doesn't guarantee people think about it\n> again as things evolve but it should at least help with getting the\n> right setting for new plumbing.\n\nI suspect we'd be about as well off with the \"don't trust the repo\"\ninternal flag. Touching the ALLOW_PAGER setup code is about as likely to\nset off red flags for the developers (or reviewers) as code that checks\nthe \"trust\" flag.\n\nForcing a whitelist on ALLOW_PAGER _is_ more likely to catch people\nadding new commands. But I don't think we actually want to add more\ncommands to the \"safe to run in a malicious repo\" list. It's already a\nslightly sketchy concept. This is really all about upload-pack and its\nexisting promises.\n\nBut ALLOW_PAGER would _just_ fix the pager issue. When we inevitably\nfind another problem spot, it won't help us there. But a global \"trust\"\nflag might.\n\nI dunno. I guess I'm OK with either approach, but it seems like the\nglobal trust flag has more room to grow.\n\n-Peff\n"},{"id":"341542","messageId":"20180313084901.41dcc541d0d0273c62a571ca@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-9-bmwill@google.com","subject":"Re: [PATCH v4 08/35] connect: discover protocol version outside of get_remote_heads","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-03-13T15:49:01Z","receivedAt":"2018-03-13T15:49:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 28 Feb 2018 15:22:25 -0800\nBrandon Williams wrote:\n\n> In order to prepare for the addition of protocol_v2 push the protocol\n> version discovery outside of 'get_remote_heads()'. This will allow for\n> keeping the logic for processing the reference advertisement for\n> protocol_v1 and protocol_v0 separate from the logic for protocol_v2.\n> \n> Signed-off-by: Brandon Williams \n\nI had one issue in version 3 which turns out to not be one, and this\npatch is unchanged from version 3, so:\n\nReviewed-by: Jonathan Tan \n\n> ---\n> builtin/fetch-pack.c | 16 +++++++++++++++-\n> builtin/send-pack.c | 17 +++++++++++++++--\n> connect.c | 27 ++++++++++-----------------\n> connect.h | 3 +++\n> remote-curl.c | 20 ++++++++++++++++++--\n> remote.h | 5 +++--\n> transport.c | 24 +++++++++++++++++++-----\n> 7 files changed, 83 insertions(+), 29 deletions(-)\n> \n> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n> index 366b9d13f..85d4faf76 100644\n> --- a/builtin/fetch-pack.c\n> +++ b/builtin/fetch-pack.c\n> @@ -4,6 +4,7 @@\n> #include \"remote.h\"\n> #include \"connect.h\"\n> #include \"sha1-array.h\"\n> +#include \"protocol.h\"\n> \n> static const char fetch_pack_usage[] =\n> \"git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] \"\n> @@ -52,6 +53,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> \tstruct fetch_pack_args args;\n> \tstruct oid_array shallow = OID_ARRAY_INIT;\n> \tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n> +\tstruct packet_reader reader;\n> \n> \tpacket_trace_identity(\"fetch-pack\");\n> \n> @@ -193,7 +195,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> \t\tif (!conn)\n> \t\t\treturn args.diag_url ? 0 : 1;\n> \t}\n> -\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n> +\n> +\tpacket_reader_init(&reader, fd[0], NULL, 0,\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> +\n> +\tswitch (discover_version(&reader)) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tget_remote_heads(&reader, &ref, 0, NULL, &shallow);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> \n> \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n> \t\t\t &shallow, pack_lockfile_ptr);\n> diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n> index fc4f0bb5f..83cb125a6 100644\n> --- a/builtin/send-pack.c\n> +++ b/builtin/send-pack.c\n> @@ -14,6 +14,7 @@\n> #include \"sha1-array.h\"\n> #include \"gpg-interface.h\"\n> #include \"gettext.h\"\n> +#include \"protocol.h\"\n> \n> static const char * const send_pack_usage[] = {\n> \tN_(\"git send-pack [--all | --mirror] [--dry-run] [--force] \"\n> @@ -154,6 +155,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n> \tint progress = -1;\n> \tint from_stdin = 0;\n> \tstruct push_cas_option cas = {0};\n> +\tstruct packet_reader reader;\n> \n> \tstruct option options[] = {\n> \t\tOPT__VERBOSITY(&verbose),\n> @@ -256,8 +258,19 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n> \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n> \t}\n> \n> -\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n> -\t\t\t &extra_have, &shallow);\n> +\tpacket_reader_init(&reader, fd[0], NULL, 0,\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> +\n> +\tswitch (discover_version(&reader)) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tget_remote_heads(&reader, &remote_refs, REF_NORMAL,\n> +\t\t\t\t &extra_have, &shallow);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> \n> \ttransport_verify_remote_names(nr_refspecs, refspecs);\n> \n> diff --git a/connect.c b/connect.c\n> index c82c90b7c..0b111e62d 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -62,7 +62,7 @@ static void die_initial_contact(int unexpected)\n> \t\t \"and the repository exists.\"));\n> }\n> \n> -static enum protocol_version discover_version(struct packet_reader *reader)\n> +enum protocol_version discover_version(struct packet_reader *reader)\n> {\n> \tenum protocol_version version = protocol_unknown_version;\n> \n> @@ -233,7 +233,7 @@ enum get_remote_heads_state {\n> /*\n> * Read all the refs from the other end\n> */\n> -struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> +struct ref **get_remote_heads(struct packet_reader *reader,\n> \t\t\t struct ref **list, unsigned int flags,\n> \t\t\t struct oid_array *extra_have,\n> \t\t\t struct oid_array *shallow_points)\n> @@ -241,24 +241,17 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \tstruct ref **orig_list = list;\n> \tint len = 0;\n> \tenum get_remote_heads_state state = EXPECTING_FIRST_REF;\n> -\tstruct packet_reader reader;\n> \tconst char *arg;\n> \n> -\tpacket_reader_init(&reader, in, src_buf, src_len,\n> -\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> -\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> -\n> -\tdiscover_version(&reader);\n> -\n> \t*list = NULL;\n> \n> \twhile (state != EXPECTING_DONE) {\n> -\t\tswitch (packet_reader_read(&reader)) {\n> +\t\tswitch (packet_reader_read(reader)) {\n> \t\tcase PACKET_READ_EOF:\n> \t\t\tdie_initial_contact(1);\n> \t\tcase PACKET_READ_NORMAL:\n> -\t\t\tlen = reader.pktlen;\n> -\t\t\tif (len > 4 && skip_prefix(reader.line, \"ERR \", &arg))\n> +\t\t\tlen = reader->pktlen;\n> +\t\t\tif (len > 4 && skip_prefix(reader->line, \"ERR \", &arg))\n> \t\t\t\tdie(\"remote error: %s\", arg);\n> \t\t\tbreak;\n> \t\tcase PACKET_READ_FLUSH:\n> @@ -270,22 +263,22 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \n> \t\tswitch (state) {\n> \t\tcase EXPECTING_FIRST_REF:\n> -\t\t\tprocess_capabilities(reader.line, &len);\n> -\t\t\tif (process_dummy_ref(reader.line)) {\n> +\t\t\tprocess_capabilities(reader->line, &len);\n> +\t\t\tif (process_dummy_ref(reader->line)) {\n> \t\t\t\tstate = EXPECTING_SHALLOW;\n> \t\t\t\tbreak;\n> \t\t\t}\n> \t\t\tstate = EXPECTING_REF;\n> \t\t\t/* fallthrough */\n> \t\tcase EXPECTING_REF:\n> -\t\t\tif (process_ref(reader.line, len, &list, flags, extra_have))\n> +\t\t\tif (process_ref(reader->line, len, &list, flags, extra_have))\n> \t\t\t\tbreak;\n> \t\t\tstate = EXPECTING_SHALLOW;\n> \t\t\t/* fallthrough */\n> \t\tcase EXPECTING_SHALLOW:\n> -\t\t\tif (process_shallow(reader.line, len, shallow_points))\n> +\t\t\tif (process_shallow(reader->line, len, shallow_points))\n> \t\t\t\tbreak;\n> -\t\t\tdie(\"protocol error: unexpected '%s'\", reader.line);\n> +\t\t\tdie(\"protocol error: unexpected '%s'\", reader->line);\n> \t\tcase EXPECTING_DONE:\n> \t\t\tbreak;\n> \t\t}\n> diff --git a/connect.h b/connect.h\n> index 01f14cdf3..cdb8979dc 100644\n> --- a/connect.h\n> +++ b/connect.h\n> @@ -13,4 +13,7 @@ extern int parse_feature_request(const char *features, const char *feature);\n> extern const char *server_feature_value(const char *feature, int *len_ret);\n> extern int url_is_local_not_ssh(const char *url);\n> \n> +struct packet_reader;\n> +extern enum protocol_version discover_version(struct packet_reader *reader);\n> +\n> #endif\n> diff --git a/remote-curl.c b/remote-curl.c\n> index 0053b0954..9f6d07683 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -1,6 +1,7 @@\n> #include \"cache.h\"\n> #include \"config.h\"\n> #include \"remote.h\"\n> +#include \"connect.h\"\n> #include \"strbuf.h\"\n> #include \"walker.h\"\n> #include \"http.h\"\n> @@ -13,6 +14,7 @@\n> #include \"credential.h\"\n> #include \"sha1-array.h\"\n> #include \"send-pack.h\"\n> +#include \"protocol.h\"\n> \n> static struct remote *remote;\n> /* always ends with a trailing slash */\n> @@ -176,8 +178,22 @@ static struct discovery *last_discovery;\n> static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n> {\n> \tstruct ref *list = NULL;\n> -\tget_remote_heads(-1, heads->buf, heads->len, &list,\n> -\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n> +\tstruct packet_reader reader;\n> +\n> +\tpacket_reader_init(&reader, -1, heads->buf, heads->len,\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> +\n> +\tswitch (discover_version(&reader)) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tget_remote_heads(&reader, &list, for_push ? REF_NORMAL : 0,\n> +\t\t\t\t NULL, &heads->shallow);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> +\n> \treturn list;\n> }\n> \n> diff --git a/remote.h b/remote.h\n> index 1f6611be2..2016461df 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -150,10 +150,11 @@ int check_ref_type(const struct ref *ref, int flags);\n> void free_refs(struct ref *ref);\n> \n> struct oid_array;\n> -extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> +struct packet_reader;\n> +extern struct ref **get_remote_heads(struct packet_reader *reader,\n> \t\t\t\t struct ref **list, unsigned int flags,\n> \t\t\t\t struct oid_array *extra_have,\n> -\t\t\t\t struct oid_array *shallow);\n> +\t\t\t\t struct oid_array *shallow_points);\n> \n> int resolve_remote_symref(struct ref *ref, struct ref *list);\n> int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);\n> diff --git a/transport.c b/transport.c\n> index 8e8779096..63c3dbab9 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -18,6 +18,7 @@\n> #include \"sha1-array.h\"\n> #include \"sigchain.h\"\n> #include \"transport-internal.h\"\n> +#include \"protocol.h\"\n> \n> static void set_upstreams(struct transport *transport, struct ref *refs,\n> \tint pretend)\n> @@ -190,13 +191,26 @@ static int connect_setup(struct transport *transport, int for_push)\n> static struct ref *get_refs_via_connect(struct transport *transport, int for_push)\n> {\n> \tstruct git_transport_data *data = transport->data;\n> -\tstruct ref *refs;\n> +\tstruct ref *refs = NULL;\n> +\tstruct packet_reader reader;\n> \n> \tconnect_setup(transport, for_push);\n> -\tget_remote_heads(data->fd[0], NULL, 0, &refs,\n> -\t\t\t for_push ? REF_NORMAL : 0,\n> -\t\t\t &data->extra_have,\n> -\t\t\t &data->shallow);\n> +\n> +\tpacket_reader_init(&reader, data->fd[0], NULL, 0,\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE |\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF);\n> +\n> +\tswitch (discover_version(&reader)) {\n> +\tcase protocol_v1:\n> +\tcase protocol_v0:\n> +\t\tget_remote_heads(&reader, &refs,\n> +\t\t\t\t for_push ? REF_NORMAL : 0,\n> +\t\t\t\t &data->extra_have,\n> +\t\t\t\t &data->shallow);\n> +\t\tbreak;\n> +\tcase protocol_unknown_version:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n> \tdata->got_remote_heads = 1;\n> \n> \treturn refs;\n> -- \n> 2.16.2.395.g2e18187dfd-goog\n> \n"},{"id":"341543","messageId":"20180313090020.036f0314549cc23d9f71ce87@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-17-bmwill@google.com","subject":"Re: [PATCH v4 16/35] transport: convert transport_get_remote_refs to take a list of ref patterns","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-03-13T16:00:20Z","receivedAt":"2018-03-13T16:00:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 28 Feb 2018 15:22:33 -0800\nBrandon Williams wrote:\n\n> Convert 'transport_get_remote_refs()' to optionally take a list of ref\n> patterns.\n> \n> Signed-off-by: Brandon Williams \n\n[snip]\n\n> -const struct ref *transport_get_remote_refs(struct transport *transport);\n> +/*\n> + * Retrieve refs from a remote.\n> + *\n> + * Optionally a list of ref patterns can be provided which can be sent to the\n> + * server (when communicating using protocol v2) to enable it to limit the ref\n> + * advertisement. Since ref filtering is done on the server's end (and only\n> + * when using protocol v2), this can return refs which don't match the provided\n> + * ref_patterns.\n> + */\n> +const struct ref *transport_get_remote_refs(struct transport *transport,\n> +\t\t\t\t\t const struct argv_array *ref_patterns);\n\nThanks for adding the documentation, but I think this should also go\ninto the commit message. For example:\n\n Teach transport_get_remote_refs() to accept a list of ref patterns,\n which will be sent to the server for use in filtering when using\n protocol v2. (This list will be ignored when not using protocol v2.)\n"},{"id":"341545","messageId":"20180313092003.109ce9254b376e7fdce338b0@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-21-bmwill@google.com","subject":"Re: [PATCH v4 20/35] upload-pack: introduce fetch server command","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-03-13T16:20:03Z","receivedAt":"2018-03-13T16:20:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 28 Feb 2018 15:22:37 -0800\nBrandon Williams wrote:\n\n> + output = *section\n> + section = (acknowledgments | packfile)\n> +\t (flush-pkt | delim-pkt)\n> +\n> + acknowledgments = PKT-LINE(\"acknowledgments\" LF)\n> +\t\t (nak | *ack)\n> +\t\t (ready)\n> + ready = PKT-LINE(\"ready\" LF)\n> + nak = PKT-LINE(\"NAK\" LF)\n> + ack = PKT-LINE(\"ACK\" SP obj-id LF)\n> +\n> + packfile = PKT-LINE(\"packfile\" LF)\n> +\t [PACKFILE]\n\nI should have noticed this earlier, but \"PACKFILE\" is not defined anywhere -\nit's probably better written as:\n\n *PKT-LINE(%x01-03 *%x00-ff)\"\n\nor something like that.\n\n> + acknowledgments section\n> +\t* Always begins with the section header \"acknowledgments\"\n> +\n> +\t* The server will respond with \"NAK\" if none of the object ids sent\n> +\t as have lines were common.\n> +\n> +\t* The server will respond with \"ACK obj-id\" for all of the\n> +\t object ids sent as have lines which are common.\n> +\n> +\t* A response cannot have both \"ACK\" lines as well as a \"NAK\"\n> +\t line.\n> +\n> +\t* The server will respond with a \"ready\" line indicating that\n> +\t the server has found an acceptable common base and is ready to\n> +\t make and send a packfile (which will be found in the packfile\n> +\t section of the same response)\n> +\n> +\t* If the client determines that it is finished with negotiations\n> +\t by sending a \"done\" line, the acknowledgments sections MUST be\n> +\t omitted from the server's response.\n> +\n> +\t* If the server has found a suitable cut point and has decided\n> +\t to send a \"ready\" line, then the server can decide to (as an\n> +\t optimization) omit any \"ACK\" lines it would have sent during\n> +\t its response. This is because the server will have already\n> +\t determined the objects it plans to send to the client and no\n> +\t further negotiation is needed.\n> +\n> +----\n> + packfile section\n> +\t* Always begins with the section header \"packfile\"\n> +\n> +\t* The transmission of the packfile begins immediately after the\n> +\t section header\n> +\n> +\t* The data transfer of the packfile is always multiplexed, using\n> +\t the same semantics of the 'side-band-64k' capability from\n> +\t protocol version 1. This means that each packet, during the\n> +\t packfile data stream, is made up of a leading 4-byte pkt-line\n> +\t length (typical of the pkt-line format), followed by a 1-byte\n> +\t stream code, followed by the actual data.\n> +\n> +\t The stream code can be one of:\n> +\t\t1 - pack data\n> +\t\t2 - progress messages\n> +\t\t3 - fatal error message just before stream aborts\n> +\n> +\t* This section is only included if the client has sent 'want'\n> +\t lines in its request and either requested that no more\n> +\t negotiation be done by sending 'done' or if the server has\n> +\t decided it has found a sufficient cut point to produce a\n> +\t packfile.\n\nFor both the sections, I think that the conditions for\ninclusion/non-inclusion (\"This section is only included if...\") should\nbe the first point.\n\n> +static void upload_pack_data_init(struct upload_pack_data *data)\n> +{\n> +\tstruct object_array wants = OBJECT_ARRAY_INIT;\n> +\tstruct oid_array haves = OID_ARRAY_INIT;\n> +\n> +\tmemset(data, 0, sizeof(*data));\n> +\tdata->wants = wants;\n> +\tdata->haves = haves;\n> +}\n\nAny reason to use a initializer function instead of a static literal?\n"},{"id":"341546","messageId":"20180313093044.69af995505d0508f8a044bff@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-28-bmwill@google.com","subject":"Re: [PATCH v4 27/35] transport-helper: introduce stateless-connect","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-03-13T16:30:44Z","receivedAt":"2018-03-13T16:30:54Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 28 Feb 2018 15:22:44 -0800\nBrandon Williams wrote:\n\n> +'stateless-connect'::\n> +\tExperimental; for internal use only.\n> +\tCan attempt to connect to a remote server for communication\n> +\tusing git's wire-protocol version 2. This establishes a\n> +\tstateless, half-duplex connection.\n> ++\n> +Supported commands: 'stateless-connect'.\n> +\n> 'push'::\n> \tCan discover remote refs and push local commits and the\n> \thistory leading up to them to new or existing remote refs.\n> @@ -136,6 +144,14 @@ Capabilities for Fetching\n> +\n> Supported commands: 'connect'.\n> \n> +'stateless-connect'::\n> +\tExperimental; for internal use only.\n> +\tCan attempt to connect to a remote server for communication\n> +\tusing git's wire-protocol version 2. This establishes a\n> +\tstateless, half-duplex connection.\n> ++\n> +Supported commands: 'stateless-connect'.\n\nI don't think we should use the term \"half-duplex\" - from a search, it\nmeans that both parties can use the wire but not simultaneously, which\nis not strictly true. Might be better to just say \"see the documentation\nfor the stateless-connect command for more information\".\n\n> +'stateless-connect' ::\n> +\tExperimental; for internal use only.\n> +\tConnects to the given remote service for communication using\n> +\tgit's wire-protocol version 2. This establishes a stateless,\n> +\thalf-duplex connection. Valid replies to this command are empty\n> +\tline (connection established), 'fallback' (no smart transport\n> +\tsupport, fall back to dumb transports) and just exiting with\n> +\terror message printed (can't connect, don't bother trying to\n> +\tfall back). After line feed terminating the positive (empty)\n> +\tresponse, the output of the service starts. Messages (both\n> +\trequest and response) must be terminated with a single flush\n> +\tpacket, allowing the remote helper to properly act as a proxy.\n> +\tAfter the connection ends, the remote helper exits.\n> ++\n> +Supported if the helper has the \"stateless-connect\" capability.\n\nI'm not sure of the relevance of \"allowing the remote helper to properly\nact as a proxy\" - this scheme does make it easier to implement proxies,\nnot for any party to start acting as one instead. I would write that\npart as:\n\n Messages (both request and response) must consist of zero or more\n PKT-LINEs, terminating in a flush packet. The client must not expect\n the server to store any state in between request-response pairs.\n\n(This covers the so-called \"half-duplex\" part and the \"stateless\" part.)\n"},{"id":"341548","messageId":"20180313093221.1d63e5212d8572fb0d078577@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-30-bmwill@google.com","subject":"Re: [PATCH v4 29/35] remote-curl: create copy of the service name","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-03-13T16:32:21Z","receivedAt":"2018-03-13T16:32:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 28 Feb 2018 15:22:46 -0800\nBrandon Williams wrote:\n\n> Make a copy of the service name being requested instead of relying on\n> the buffer pointed to by the passed in 'const char *' to remain\n> unchanged.\n> \n> Currently, all service names are string constants, but a subsequent\n> patch will introduce service names from external sources.\n> \n> Signed-off-by: Brandon Williams \n\nOnce again,\n\nReviewed-by: Jonathan Tan \n"},{"id":"341549","messageId":"20180313093348.34f10e0ca4e14a44a62a5049@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-32-bmwill@google.com","subject":"Re: [PATCH v4 31/35] http: allow providing extra headers for http requests","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-03-13T16:33:48Z","receivedAt":"2018-03-13T16:34:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 28 Feb 2018 15:22:48 -0800\nBrandon Williams wrote:\n\n> Add a way for callers to request that extra headers be included when\n> making http requests.\n> \n> Signed-off-by: Brandon Williams \n\nLikewise,\n\nReviewed-by: Jonathan Tan \n"},{"id":"341550","messageId":"20180313093546.6ae2961e8fbb9e8bf7e590c3@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-36-bmwill@google.com","subject":"Re: [PATCH v4 35/35] remote-curl: don't request v2 when pushing","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-03-13T16:35:46Z","receivedAt":"2018-03-13T16:35:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 28 Feb 2018 15:22:52 -0800\nBrandon Williams wrote:\n\n> In order to be able to ship protocol v2 with only supporting fetch, we\n> need clients to not issue a request to use protocol v2 when pushing\n> (since the client currently doesn't know how to push using protocol v2).\n> This allows a client to have protocol v2 configured in\n> `protocol.version` and take advantage of using v2 for fetch and falling\n> back to using v0 when pushing while v2 for push is being designed.\n> \n> We could run into issues if we didn't fall back to protocol v2 when\n> pushing right now. This is because currently a server will ignore a request to\n> use v2 when contacting the 'receive-pack' endpoint and fall back to\n> using v0, but when push v2 is rolled out to servers, the 'receive-pack'\n> endpoint will start responding using v2. So we don't want to get into a\n> state where a client is requesting to push with v2 before they actually\n> know how to push using v2.\n> \n> Signed-off-by: Brandon Williams \n\nI noticed that my review comments have been addressed, so:\n\nReviewed-by: Jonathan Tan \n"},{"id":"341551","messageId":"20180313094022.6cbe51d765e2eec61e0cd04c@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-5-bmwill@google.com","subject":"Re: [PATCH v4 04/35] upload-pack: convert to a builtin","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-03-13T16:40:22Z","receivedAt":"2018-03-13T16:40:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 28 Feb 2018 15:22:21 -0800\nBrandon Williams wrote:\n\n> In order to allow for code sharing with the server-side of fetch in\n> protocol-v2 convert upload-pack to be a builtin.\n> \n> Signed-off-by: Brandon Williams \n\nI suggested updating the commit message in my previous review [1], but I\nunderstand that my comment might have been lost in the ensuing long\ndiscussion.\n\n[1] https://public-inbox.org/git/20180221134422.2386e1aca39fe673235590e7@google.com/\n"},{"id":"341575","messageId":"20180313123540.d6e416220329bdc172472dc6@google.com","threadId":"47524","inReplyTo":"20180228232252.102167-2-bmwill@google.com","subject":"Re: [PATCH v4 01/35] pkt-line: introduce packet_read_with_status","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-03-13T19:35:40Z","receivedAt":"2018-03-13T19:35:51Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 28 Feb 2018 15:22:18 -0800\nBrandon Williams wrote:\n\n> +\tif (len < 0) {\n> \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n> -\tif (!len) {\n> +\t} else if (!len) {\n> \t\tpacket_trace(\"0000\", 4, 0);\n> -\t\treturn 0;\n> +\t\treturn PACKET_READ_FLUSH;\n> +\t} else if (len < 4) {\n> +\t\tdie(\"protocol error: bad line length %d\", len);\n> \t}\n> +\n> \tlen -= 4;\n> -\tif (len >= size)\n> +\tif ((unsigned)len >= size)\n> \t\tdie(\"protocol error: bad line length %d\", len);\n\nThe cast to unsigned is safe, since len was at least 4 before \"len -=\n4\". I can't think of a better way to write this to make that more\nobvious, though.\n\n> +/*\n> + * Read a packetized line into a buffer like the 'packet_read()' function but\n> + * returns an 'enum packet_read_status' which indicates the status of the read.\n> + * The number of bytes read will be assigined to *pktlen if the status of the\n> + * read was 'PACKET_READ_NORMAL'.\n> + */\n> +enum packet_read_status {\n> +\tPACKET_READ_EOF = -1,\n> +\tPACKET_READ_NORMAL,\n> +\tPACKET_READ_FLUSH,\n> +};\n> +enum packet_read_status packet_read_with_status(int fd, char **src_buffer,\n> +\t\t\t\t\t\tsize_t *src_len, char *buffer,\n> +\t\t\t\t\t\tunsigned size, int *pktlen,\n> +\t\t\t\t\t\tint options);\n\njrnieder said in [1], referring to the definition of enum\npacket_read_status:\n\n> nit: do any callers treat the return value as a number? It would be\n> less magical if the numbering were left to the compiler (0, 1, 2).\n\nI checked the result of the entire patch set and the only callers seem\nto be packet_read() (modified in this patch) and the\nsoon-to-be-introduced packet_reader_read(). So not only can the\nnumbering be left to the compiler, this function can (and should) be\nmarked static as well (and the enum definition moved to .c), since I\nthink that future development should be encouraged to use packet_reader.\n\nThe commit message would also thus need to be rewritten, since this\nbecomes more of a refactoring into a function with a more precisely\nspecified return type, to be used both by the existing packet_read() and\na soon-to-be-introduced packet_reader_read().\n\n[1] https://public-inbox.org/git/20180213002554.GA42272@aiede.svl.corp.google.com/\n"},{"id":"341576","messageId":"20180313195044.GA7638@google.com","threadId":"47524","inReplyTo":"20180313094022.6cbe51d765e2eec61e0cd04c@google.com","subject":"Re: [PATCH v4 04/35] upload-pack: convert to a builtin","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-13T19:50:44Z","receivedAt":"2018-03-13T19:50:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/13, Jonathan Tan wrote:\n> On Wed, 28 Feb 2018 15:22:21 -0800\n> Brandon Williams wrote:\n> \n> > In order to allow for code sharing with the server-side of fetch in\n> > protocol-v2 convert upload-pack to be a builtin.\n> > \n> > Signed-off-by: Brandon Williams \n> \n> I suggested updating the commit message in my previous review [1], but I\n> understand that my comment might have been lost in the ensuing long\n> discussion.\n> \n> [1] https://public-inbox.org/git/20180221134422.2386e1aca39fe673235590e7@google.com/\n\nYour suggested change to the commit msg isn't quite accurate as you can\nalready run \"git-upload-pack --help\" today.\n\n-- \nBrandon Williams\n"},{"id":"341577","messageId":"20180313195220.GB7638@google.com","threadId":"47524","inReplyTo":"20180313123540.d6e416220329bdc172472dc6@google.com","subject":"Re: [PATCH v4 01/35] pkt-line: introduce packet_read_with_status","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-13T19:52:20Z","receivedAt":"2018-03-13T19:52:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/13, Jonathan Tan wrote:\n> On Wed, 28 Feb 2018 15:22:18 -0800\n> Brandon Williams wrote:\n> \n> > +\tif (len < 0) {\n> > \t\tdie(\"protocol error: bad line length character: %.4s\", linelen);\n> > -\tif (!len) {\n> > +\t} else if (!len) {\n> > \t\tpacket_trace(\"0000\", 4, 0);\n> > -\t\treturn 0;\n> > +\t\treturn PACKET_READ_FLUSH;\n> > +\t} else if (len < 4) {\n> > +\t\tdie(\"protocol error: bad line length %d\", len);\n> > \t}\n> > +\n> > \tlen -= 4;\n> > -\tif (len >= size)\n> > +\tif ((unsigned)len >= size)\n> > \t\tdie(\"protocol error: bad line length %d\", len);\n> \n> The cast to unsigned is safe, since len was at least 4 before \"len -=\n> 4\". I can't think of a better way to write this to make that more\n> obvious, though.\n> \n> > +/*\n> > + * Read a packetized line into a buffer like the 'packet_read()' function but\n> > + * returns an 'enum packet_read_status' which indicates the status of the read.\n> > + * The number of bytes read will be assigined to *pktlen if the status of the\n> > + * read was 'PACKET_READ_NORMAL'.\n> > + */\n> > +enum packet_read_status {\n> > +\tPACKET_READ_EOF = -1,\n> > +\tPACKET_READ_NORMAL,\n> > +\tPACKET_READ_FLUSH,\n> > +};\n> > +enum packet_read_status packet_read_with_status(int fd, char **src_buffer,\n> > +\t\t\t\t\t\tsize_t *src_len, char *buffer,\n> > +\t\t\t\t\t\tunsigned size, int *pktlen,\n> > +\t\t\t\t\t\tint options);\n> \n> jrnieder said in [1], referring to the definition of enum\n> packet_read_status:\n> \n> > nit: do any callers treat the return value as a number? It would be\n> > less magical if the numbering were left to the compiler (0, 1, 2).\n\nyeah i'll do this.\n\n> \n> I checked the result of the entire patch set and the only callers seem\n> to be packet_read() (modified in this patch) and the\n> soon-to-be-introduced packet_reader_read(). So not only can the\n> numbering be left to the compiler, this function can (and should) be\n> marked static as well (and the enum definition moved to .c), since I\n> think that future development should be encouraged to use packet_reader.\n\nThe enum definition can't be moved as its needed outside this file.\n\n> \n> The commit message would also thus need to be rewritten, since this\n> becomes more of a refactoring into a function with a more precisely\n> specified return type, to be used both by the existing packet_read() and\n> a soon-to-be-introduced packet_reader_read().\n> \n> [1] https://public-inbox.org/git/20180213002554.GA42272@aiede.svl.corp.google.com/\n\n-- \nBrandon Williams\n"},{"id":"341587","messageId":"20180313212312.GC7638@google.com","threadId":"47524","inReplyTo":"20180305202813.GA5953@sigill.intra.peff.net","subject":"Re: [PATCH v4 13/35] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-13T21:23:12Z","receivedAt":"2018-03-13T21:23:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/05, Jeff King wrote:\n> On Mon, Mar 05, 2018 at 10:21:55AM -0800, Brandon Williams wrote:\n> \n> > > Hmm, so this would accept stuff like \"refs/heads/*/foo\" but quietly\n> > > ignore the \"/foo\" part.\n> > \n> > Yeah that's true...this should probably not do that. Since\n> > \"refs/heads/*/foo\" violates what the spec is, really this should error\n> > out as an invalid pattern.\n> \n> Yeah, that would be better, I think.\n> \n> > > It also accepts \"refs/h*\" to get \"refs/heads\" and \"refs/hello\". I think\n> > > it's worth going for the most-restrictive thing to start with, since\n> > > that enables a lot more server operations without worrying about\n> > > breaking compatibility.\n> > \n> > And just to clarify what do you see as being the most-restrictive case\n> > of patterns that would should use?\n> \n> I mean only accepting \"*\" at a \"/\" boundary (or just allowing a trailing\n> slash to imply recursion, like \"refs/heads/\", or even just always\n> assuming recursion to allow \"refs/heads\").\n\nFor simplicity I'll change ref-patterns to be ref-prefixes where\na ref must start_with() one of the provided ref-prefixes. Clients won't\nsend '*'s either but can send everything upto but not including the '*'\nas a prefix.\n\n-- \nBrandon Williams\n"},{"id":"341588","messageId":"20180313212710.GD7638@google.com","threadId":"47524","inReplyTo":"xmqqmuzqb2ph.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 13/35] ls-refs: introduce ls-refs server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-13T21:27:10Z","receivedAt":"2018-03-13T21:27:17Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/02, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > + ls-refs\n> > +---------\n> > +\n> > +`ls-refs` is the command used to request a reference advertisement in v2.\n> > +Unlike the current reference advertisement, ls-refs takes in arguments\n> > +which can be used to limit the refs sent from the server.\n> \n> OK.\n> \n> > +Additional features not supported in the base command will be advertised\n> > +as the value of the command in the capability advertisement in the form\n> > +of a space separated list of features, e.g. \"=\n> > +\".\n> \n> Doesn't this explain the general convention that applies to any\n> command, not just ls-refs command? As a part of ls-refs section,\n> in the above explanation is always a constant \"ls-refs\",\n> right?\n> \n> It is a bit unclear how in the above description are\n> related to \"arguments\" in the following paragraph. Do the server\n> that can show symref and peeled tags and that can limit the output\n> with ref-pattern advertise these three as supported features, i.e.\n> \n> \tls-refs=symrefs peel ref-pattern\n> \n> or something? Would there a case where a \"feature\" does not\n> correspond 1:1 to an argument to the command, and if so how would\n> the server and the client negotiate use of such a feature?\n\nI mention earlier in the document that the values of each capability are\nto be defined by the capability itself, so I'm just documenting what the\nvalue advertised means. And a feature could mean a couple things and\ndoesn't necessarily mean it affects the arguments which can be provided,\nand it definitely doesn't mean that each argument that can be provided\nmust be advertised as a feature. If you look at the patch that\nintroduces shallow, the shallow feature adds lots of arguments that a\nclient can that use in its request.\n\n> \n> > + ref-pattern \n> > +\tWhen specified, only references matching one of the provided\n> > +\tpatterns are displayed. A pattern is either a valid refname\n> > +\t(e.g. refs/heads/master), in which a ref must match the pattern\n> > +\texactly, or a prefix of a ref followed by a single '*' wildcard\n> > +\tcharacter (e.g. refs/heads/*), in which a ref must have a prefix\n> > +\tequal to the pattern up to the wildcard character.\n> \n> I thought the recent concensus was left-anchored prefix match that\n> honors /-directory boundary, i.e. no explicit asterisk and just\n> saying \"refs/heads\" is enough to match \"refs/heads\" itself and\n> \"refs/heads/master\" but not \"refs/headscarf\"?\n\nI don't think there was a consensus at the time, but in the next\nrevision I'll have them be prefixes instead of containing wildcards.\n\n-- \nBrandon Williams\n"},{"id":"341589","messageId":"20180313213542.GE7638@google.com","threadId":"47524","inReplyTo":"xmqqzi3qb3if.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 12/35] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-13T21:35:42Z","receivedAt":"2018-03-13T21:35:50Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/02, Junio C Hamano wrote:\n> Brandon Williams writes:\n> > +static int is_command(const char *key, struct protocol_capability **command)\n> > +{\n> > +\tconst char *out;\n> > +\n> > +\tif (skip_prefix(key, \"command=\", &out)) {\n> > +\t\tstruct protocol_capability *cmd = get_capability(out);\n> > +\n> > +\t\tif (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command)\n> > +\t\t\tdie(\"invalid command '%s'\", out);\n> > +\t\tif (*command)\n> > +\t\t\tdie(\"command already requested\");\n> \n> Shouldn't these two checks that lead to die the other way around?\n> When they give us \"command=frotz\" and we already have *command, it\n> would be an error whether we understand 'frotz' or not.\n> \n> Who are the target audience of these \"die\"? Are they meant to be\n> communicated back to the other side of the connection, or are they\n> only to be sent to the \"server log\"?\n> \n> The latter one may want to say what two conflicting commands are in\n> the log message, perhaps?\n\nYeah I'll switch the order of these checks as well as print out the two\ncommands requested for better logging.\n\n> \n> > +\t\t*command = cmd;\n> \n\n-- \nBrandon Williams\n"},{"id":"341591","messageId":"20180313214024.GF7638@google.com","threadId":"47524","inReplyTo":"xmqq4llyciq8.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 12/35] serve: introduce git-serve","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-13T21:40:24Z","receivedAt":"2018-03-13T21:40:30Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/02, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > + Capabilities\n> > +~~~~~~~~~~~~~~\n> > +\n> > +There are two different types of capabilities: normal capabilities,\n> > +which can be used to to convey information or alter the behavior of a\n> > +request, and commands, which are the core actions that a client wants to\n> > +perform (fetch, push, etc).\n> > +\n> > +All commands must only last a single round and be stateless from the\n> > +perspective of the server side. All state MUST be retained and managed\n> > +by the client process. This permits simple round-robin load-balancing\n> > +on the server side, without needing to worry about state management.\n> > +\n> > +Clients MUST NOT require state management on the server side in order to\n> > +function correctly.\n> \n> This somehow feels a bit too HTTP centric worldview that potentially\n> may penalize those who do not mind stateful services.\n\nIt's meant to be that way so that we don't run into the same issue we\nhave with the current HTTP transport. Though I've decided to loosen\nthis slightly by making protocol v2 stateless by default unless a\ncapability is advertised and requested by the client indicating that\nstate can be maintained by the server. That leaves the door open for\nadding state later for transports which have full-duplex connections\nwhile still requiring that stateless is designed first. I'm kind of\nhoping we never need to add state to the protocol because hopefully we\ncan figure out how to improve negotiation as a whole.\n\n> \n> > + agent\n> > +-------\n> > +\n> > +The server can advertise the `agent` capability with a value `X` (in the\n> > +form `agent=X`) to notify the client that the server is running version\n> > +`X`. The client may optionally send its own agent string by including\n> > +the `agent` capability with a value `Y` (in the form `agent=Y`) in its\n> > +request to the server (but it MUST NOT do so if the server did not\n> > +advertise the agent capability).\n> \n> Are there different degrees of permissiveness between \"The server\n> CAN\" and \"The client MAY\" above, or is the above paragraph merely\n> being fuzzy?\n\nI don't think so? I believe I ripped this from the existing description\nof the agent capability from the current protocol.\n\n-- \nBrandon Williams\n"},{"id":"341595","messageId":"20180313214925.GG7638@google.com","threadId":"47524","inReplyTo":"20180313092003.109ce9254b376e7fdce338b0@google.com","subject":"Re: [PATCH v4 20/35] upload-pack: introduce fetch server command","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-13T21:49:25Z","receivedAt":"2018-03-13T21:49:33Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/13, Jonathan Tan wrote:\n> On Wed, 28 Feb 2018 15:22:37 -0800\n> Brandon Williams wrote:\n> \n> > + output = *section\n> > + section = (acknowledgments | packfile)\n> > +\t (flush-pkt | delim-pkt)\n> > +\n> > + acknowledgments = PKT-LINE(\"acknowledgments\" LF)\n> > +\t\t (nak | *ack)\n> > +\t\t (ready)\n> > + ready = PKT-LINE(\"ready\" LF)\n> > + nak = PKT-LINE(\"NAK\" LF)\n> > + ack = PKT-LINE(\"ACK\" SP obj-id LF)\n> > +\n> > + packfile = PKT-LINE(\"packfile\" LF)\n> > +\t [PACKFILE]\n> \n> I should have noticed this earlier, but \"PACKFILE\" is not defined anywhere -\n> it's probably better written as:\n> \n> *PKT-LINE(%x01-03 *%x00-ff)\"\n> \n> or something like that.\n\nI'll document it as you described.\n\n> \n> > + acknowledgments section\n> > +\t* Always begins with the section header \"acknowledgments\"\n> > +\n> > +\t* The server will respond with \"NAK\" if none of the object ids sent\n> > +\t as have lines were common.\n> > +\n> > +\t* The server will respond with \"ACK obj-id\" for all of the\n> > +\t object ids sent as have lines which are common.\n> > +\n> > +\t* A response cannot have both \"ACK\" lines as well as a \"NAK\"\n> > +\t line.\n> > +\n> > +\t* The server will respond with a \"ready\" line indicating that\n> > +\t the server has found an acceptable common base and is ready to\n> > +\t make and send a packfile (which will be found in the packfile\n> > +\t section of the same response)\n> > +\n> > +\t* If the client determines that it is finished with negotiations\n> > +\t by sending a \"done\" line, the acknowledgments sections MUST be\n> > +\t omitted from the server's response.\n> > +\n> > +\t* If the server has found a suitable cut point and has decided\n> > +\t to send a \"ready\" line, then the server can decide to (as an\n> > +\t optimization) omit any \"ACK\" lines it would have sent during\n> > +\t its response. This is because the server will have already\n> > +\t determined the objects it plans to send to the client and no\n> > +\t further negotiation is needed.\n> > +\n> > +----\n> > + packfile section\n> > +\t* Always begins with the section header \"packfile\"\n> > +\n> > +\t* The transmission of the packfile begins immediately after the\n> > +\t section header\n> > +\n> > +\t* The data transfer of the packfile is always multiplexed, using\n> > +\t the same semantics of the 'side-band-64k' capability from\n> > +\t protocol version 1. This means that each packet, during the\n> > +\t packfile data stream, is made up of a leading 4-byte pkt-line\n> > +\t length (typical of the pkt-line format), followed by a 1-byte\n> > +\t stream code, followed by the actual data.\n> > +\n> > +\t The stream code can be one of:\n> > +\t\t1 - pack data\n> > +\t\t2 - progress messages\n> > +\t\t3 - fatal error message just before stream aborts\n> > +\n> > +\t* This section is only included if the client has sent 'want'\n> > +\t lines in its request and either requested that no more\n> > +\t negotiation be done by sending 'done' or if the server has\n> > +\t decided it has found a sufficient cut point to produce a\n> > +\t packfile.\n> \n> For both the sections, I think that the conditions for\n> inclusion/non-inclusion (\"This section is only included if...\") should\n> be the first point.\n> \n> > +static void upload_pack_data_init(struct upload_pack_data *data)\n> > +{\n> > +\tstruct object_array wants = OBJECT_ARRAY_INIT;\n> > +\tstruct oid_array haves = OID_ARRAY_INIT;\n> > +\n> > +\tmemset(data, 0, sizeof(*data));\n> > +\tdata->wants = wants;\n> > +\tdata->haves = haves;\n> > +}\n> \n> Any reason to use a initializer function instead of a static literal?\n\nIts much cleaner and easier to read than it was when i was using an\ninitializer.\n\n-- \nBrandon Williams\n"},{"id":"341645","messageId":"20180314173652.GA135110@google.com","threadId":"47524","inReplyTo":"20180313093044.69af995505d0508f8a044bff@google.com","subject":"Re: [PATCH v4 27/35] transport-helper: introduce stateless-connect","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-03-14T17:36:52Z","receivedAt":"2018-03-14T17:37:00Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 03/13, Jonathan Tan wrote:\n> On Wed, 28 Feb 2018 15:22:44 -0800\n> Brandon Williams wrote:\n> \n> > +'stateless-connect'::\n> > +\tExperimental; for internal use only.\n> > +\tCan attempt to connect to a remote server for communication\n> > +\tusing git's wire-protocol version 2. This establishes a\n> > +\tstateless, half-duplex connection.\n> > ++\n> > +Supported commands: 'stateless-connect'.\n> > +\n> > 'push'::\n> > \tCan discover remote refs and push local commits and the\n> > \thistory leading up to them to new or existing remote refs.\n> > @@ -136,6 +144,14 @@ Capabilities for Fetching\n> > +\n> > Supported commands: 'connect'.\n> > \n> > +'stateless-connect'::\n> > +\tExperimental; for internal use only.\n> > +\tCan attempt to connect to a remote server for communication\n> > +\tusing git's wire-protocol version 2. This establishes a\n> > +\tstateless, half-duplex connection.\n> > ++\n> > +Supported commands: 'stateless-connect'.\n> \n> I don't think we should use the term \"half-duplex\" - from a search, it\n> means that both parties can use the wire but not simultaneously, which\n> is not strictly true. Might be better to just say \"see the documentation\n> for the stateless-connect command for more information\".\n> \n> > +'stateless-connect' ::\n> > +\tExperimental; for internal use only.\n> > +\tConnects to the given remote service for communication using\n> > +\tgit's wire-protocol version 2. This establishes a stateless,\n> > +\thalf-duplex connection. Valid replies to this command are empty\n> > +\tline (connection established), 'fallback' (no smart transport\n> > +\tsupport, fall back to dumb transports) and just exiting with\n> > +\terror message printed (can't connect, don't bother trying to\n> > +\tfall back). After line feed terminating the positive (empty)\n> > +\tresponse, the output of the service starts. Messages (both\n> > +\trequest and response) must be terminated with a single flush\n> > +\tpacket, allowing the remote helper to properly act as a proxy.\n> > +\tAfter the connection ends, the remote helper exits.\n> > ++\n> > +Supported if the helper has the \"stateless-connect\" capability.\n> \n> I'm not sure of the relevance of \"allowing the remote helper to properly\n> act as a proxy\" - this scheme does make it easier to implement proxies,\n> not for any party to start acting as one instead. I would write that\n> part as:\n> \n> Messages (both request and response) must consist of zero or more\n> PKT-LINEs, terminating in a flush packet. The client must not expect\n> the server to store any state in between request-response pairs.\n> \n> (This covers the so-called \"half-duplex\" part and the \"stateless\" part.)\n\nThanks for helping wordsmith this, I'll update the docs based on these\nsuggestions.\n\n-- \nBrandon Williams\n"}]}