{"thread":{"id":"21220","subject":"[RFC PATCH v2 00/16] Return of smart HTTP","startedAt":"2009-10-13T02:24:59Z","lastAt":"2009-10-13T21:51:31Z","messageCount":38,"participants":["Shawn O. Pearce","eduard stefan","Daniel Barkalow","Mike Hommey","Junio C Hamano","Johannes Sixt","Felipe Contreras","Johannes Schindelin","Jakub Narebski"],"isPatch":true,"patchVersion":2,"patchTotal":16},"messages":[{"id":"124786","messageId":"1255400715-10508-1-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":null,"subject":"[RFC PATCH v2 00/16] Return of smart HTTP","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:24:59Z","receivedAt":"2009-10-13T02:24:59Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The series has gotten a lot larger since my last posting, but I have\nwhat appears to be a fully working client *AND* server implementation\nfor both fetch and push, and the client should be supporting deeping\nshallow repositories over the smart variant of HTTP.\n\nI've dropped the documentation patch from the series for now as\nI have quite a few edits queued up from folks in the last round\n(thanks for those!) that I have not yet applied.  So there is no\npoint in sending that particular patch again.\n\nThis series is still lacking:\n\n* The HTTP protocol documentation\n* Manual page for git-http-backend\n* Tests for the smart http transport code (existing tests pass)\n* ack from the CC'd maintainers  :-)\n\nFor those who are really curious, this applies to 'master' and I'm\nrunning my local server under Apache with this as my configuration:\n\n -- httpd.conf <8--\n LoadModule cgi_module /usr/libexec/apache2/mod_cgi.so\n LoadModule alias_module /usr/libexec/apache2/mod_alias.so\n LoadModule env_module /usr/libexec/apache2/mod_env.so\n\n Listen 127.0.0.1:8079\n\n ServerRoot /home/spearce/test\n ErrorLog error_log\n PidFile httpd.pid\n LockFile httpd.lock\n\n <IfModule mime_module>\n TypesConfig /etc/mime.types\n </IfModule>\n\n DocumentRoot /home/spearce/test/www\n\n SetEnv GIT_EXEC_PATH /home/spearce/cgit\n ScriptAlias /git/ /home/spearce/cgit/git-http-backend/\n --<8--\n\n $ httpd -f httpd.conf\n $ git clone --bare foo.git /home/spearce/test/www/test.git\n\n URL is ... http://localhost:8070/git/test.git\n\n\nShawn O. Pearce (16):\n  pkt-line: Add strbuf based functions\n  pkt-line: Make packet_read_line easier to debug\n  fetch-pack: Use a strbuf to compose the want list\n  Move \"get_ack()\" back to fetch-pack\n  Add multi_ack_2 capability to fetch-pack/upload-pack\n  remote-curl: Refactor walker initialization\n  remote-helpers: Fetch more than one ref in a batch\n  remote-helpers: Support custom transport options\n  Move WebDAV HTTP push under remote-curl\n  Git-aware CGI to provide dumb HTTP transport\n  Add one shot RPC options to upload-pack, receive-pack\n  Smart fetch and push over HTTP: server side\n  Discover refs via smart HTTP server when available\n  Smart push over HTTP: client side\n  Smart fetch over HTTP: client side\n  Smart HTTP fetch: gzip requests\n\n .gitignore                           |    1 +\n Documentation/config.txt             |    8 +\n Documentation/git-remote-helpers.txt |   79 ++++\n Makefile                             |    1 +\n builtin-fetch-pack.c                 |  210 ++++++++--\n builtin-receive-pack.c               |   26 +-\n builtin-send-pack.c                  |  116 +++++-\n cache.h                              |    1 -\n commit.c                             |   10 +-\n commit.h                             |    2 +-\n connect.c                            |   21 -\n fetch-pack.h                         |    3 +-\n http-backend.c                       |  451 +++++++++++++++++++++\n http-push.c                          |   43 ++-\n pkt-line.c                           |   83 ++++-\n pkt-line.h                           |    4 +\n remote-curl.c                        |  729 +++++++++++++++++++++++++++++++--\n send-pack.h                          |    3 +-\n sideband.c                           |   11 +-\n transport-helper.c                   |  264 ++++++++++++-\n transport.c                          |   32 +--\n upload-pack.c                        |   71 +++-\n 22 files changed, 1955 insertions(+), 214 deletions(-)\n create mode 100644 http-backend.c\n"},{"id":"124787","messageId":"1255400715-10508-2-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 01/16] pkt-line: Add strbuf based functions","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:00Z","receivedAt":"2009-10-13T02:25:00Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"These routines help to work with pkt-line values inside of a strbuf,\npermitting simple formatting of buffered network messages.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n pkt-line.c |   81 +++++++++++++++++++++++++++++++++++++++++++++++++++---------\n pkt-line.h |    4 +++\n 2 files changed, 73 insertions(+), 12 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex b691abe..2333d96 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -42,17 +42,19 @@ void packet_flush(int fd)\n \tsafe_write(fd, \"0000\", 4);\n }\n \n+void packet_buf_flush(struct strbuf *buf)\n+{\n+\tstrbuf_add(buf, \"0000\", 4);\n+}\n+\n #define hex(a) (hexchar[(a) & 15])\n-void packet_write(int fd, const char *fmt, ...)\n+static char buffer[1000];\n+static unsigned format_packet(const char *fmt, va_list args)\n {\n-\tstatic char buffer[1000];\n \tstatic char hexchar[] = \"0123456789abcdef\";\n-\tva_list args;\n \tunsigned n;\n \n-\tva_start(args, fmt);\n \tn = vsnprintf(buffer + 4, sizeof(buffer) - 4, fmt, args);\n-\tva_end(args);\n \tif (n >= sizeof(buffer)-4)\n \t\tdie(\"protocol error: impossibly long line\");\n \tn += 4;\n@@ -60,9 +62,31 @@ void packet_write(int fd, const char *fmt, ...)\n \tbuffer[1] = hex(n >> 8);\n \tbuffer[2] = hex(n >> 4);\n \tbuffer[3] = hex(n);\n+\treturn n;\n+}\n+\n+void packet_write(int fd, const char *fmt, ...)\n+{\n+\tva_list args;\n+\tunsigned n;\n+\n+\tva_start(args, fmt);\n+\tn = format_packet(fmt, args);\n+\tva_end(args);\n \tsafe_write(fd, buffer, n);\n }\n \n+void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n+{\n+\tva_list args;\n+\tunsigned n;\n+\n+\tva_start(args, fmt);\n+\tn = format_packet(fmt, args);\n+\tva_end(args);\n+\tstrbuf_add(buf, buffer, n);\n+}\n+\n static void safe_read(int fd, void *buffer, unsigned size)\n {\n \tssize_t ret = read_in_full(fd, buffer, size);\n@@ -72,15 +96,11 @@ static void safe_read(int fd, void *buffer, unsigned size)\n \t\tdie(\"The remote end hung up unexpectedly\");\n }\n \n-int packet_read_line(int fd, char *buffer, unsigned size)\n+static int packet_length(unsigned *ret_len, const char *linelen)\n {\n \tint n;\n-\tunsigned len;\n-\tchar linelen[4];\n-\n-\tsafe_read(fd, linelen, 4);\n+\tunsigned len = 0;\n \n-\tlen = 0;\n \tfor (n = 0; n < 4; n++) {\n \t\tunsigned char c = linelen[n];\n \t\tlen <<= 4;\n@@ -96,8 +116,20 @@ int packet_read_line(int fd, char *buffer, unsigned size)\n \t\t\tlen += c - 'A' + 10;\n \t\t\tcontinue;\n \t\t}\n-\t\tdie(\"protocol error: bad line length character\");\n+\t\treturn -1;\n \t}\n+\t*ret_len = len;\n+\treturn 0;\n+}\n+\n+int packet_read_line(int fd, char *buffer, unsigned size)\n+{\n+\tunsigned len;\n+\tchar linelen[4];\n+\n+\tsafe_read(fd, linelen, 4);\n+\tif (packet_length(&len, linelen))\n+\t\tdie(\"protocol error: bad line length character\");\n \tif (!len)\n \t\treturn 0;\n \tlen -= 4;\n@@ -107,3 +139,28 @@ int packet_read_line(int fd, char *buffer, unsigned size)\n \tbuffer[len] = 0;\n \treturn len;\n }\n+\n+int packet_get_line(struct strbuf *out,\n+\tchar **src_buf, size_t *src_len)\n+{\n+\tunsigned len;\n+\n+\tif (*src_len < 4 || packet_length(&len, *src_buf))\n+\t\treturn -1;\n+\tif (!len) {\n+\t\t*src_buf += 4;\n+\t\t*src_len -= 4;\n+\t\treturn 0;\n+\t}\n+\tif (*src_len < len)\n+\t\treturn -2;\n+\n+\t*src_buf += 4;\n+\t*src_len -= 4;\n+\tlen -= 4;\n+\n+\tstrbuf_add(out, *src_buf, len);\n+\t*src_buf += len;\n+\t*src_len -= len;\n+\treturn len;\n+}\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 9df653f..1e5dcfe 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -2,14 +2,18 @@\n #define PKTLINE_H\n \n #include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n \n /*\n  * Silly packetized line writing interface\n  */\n void packet_flush(int fd);\n void packet_write(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+void packet_buf_flush(struct strbuf *buf);\n+void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n \n int packet_read_line(int fd, char *buffer, unsigned size);\n+int packet_get_line(struct strbuf *out, char **src_buf, size_t *src_len);\n ssize_t safe_write(int, const void *, ssize_t);\n \n #endif\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124788","messageId":"1255400715-10508-3-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 02/16] pkt-line: Make packet_read_line easier to debug","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:01Z","receivedAt":"2009-10-13T02:25:01Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When there is an error parsing the 4 byte length component we now\nNUL terminate the string and display it as part of the die message,\nthis may hint as to what data was misunderstood by the application.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n pkt-line.c |    8 +++++---\n 1 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 2333d96..350f173 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -125,11 +125,13 @@ static int packet_length(unsigned *ret_len, const char *linelen)\n int packet_read_line(int fd, char *buffer, unsigned size)\n {\n \tunsigned len;\n-\tchar linelen[4];\n+\tchar linelen[5];\n \n \tsafe_read(fd, linelen, 4);\n-\tif (packet_length(&len, linelen))\n-\t\tdie(\"protocol error: bad line length character\");\n+\tif (packet_length(&len, linelen)) {\n+\t\tlinelen[4] = '\\0';\n+\t\tdie(\"protocol error: bad line length character: %s\", linelen);\n+\t}\n \tif (!len)\n \t\treturn 0;\n \tlen -= 4;\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124791","messageId":"1255400715-10508-4-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 03/16] fetch-pack: Use a strbuf to compose the want list","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:02Z","receivedAt":"2009-10-13T02:25:02Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This change is being offered as a refactoring to make later\ncommits in the smart HTTP series easier.\n\nBy changing the enabled capabilities to be formatted in a strbuf\nit is easier to add a new capability to the set of supported\ncapabilities.\n\nBy formatting the want portion of the request into a strbuf and\nwriting it as a whole block we can later decide to hold onto\nthe req_buf (instead of releasing it) to recycle in stateless\ncommunications.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n builtin-fetch-pack.c |   52 ++++++++++++++++++++++++++++++++-----------------\n commit.c             |   10 +++-----\n commit.h             |    2 +-\n 3 files changed, 39 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex 629735f..783c2b0 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -165,6 +165,7 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tconst unsigned char *sha1;\n \tunsigned in_vain = 0;\n \tint got_continue = 0;\n+\tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (marked)\n \t\tfor_each_ref(clear_marks, NULL);\n@@ -175,6 +176,7 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\n \t\tunsigned char *remote = refs->old_sha1;\n+\t\tconst char *remote_hex;\n \t\tstruct object *o;\n \n \t\t/*\n@@ -192,27 +194,36 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tif (!fetching)\n-\t\t\tpacket_write(fd[1], \"want %s%s%s%s%s%s%s%s\\n\",\n-\t\t\t\t     sha1_to_hex(remote),\n-\t\t\t\t     (multi_ack ? \" multi_ack\" : \"\"),\n-\t\t\t\t     (use_sideband == 2 ? \" side-band-64k\" : \"\"),\n-\t\t\t\t     (use_sideband == 1 ? \" side-band\" : \"\"),\n-\t\t\t\t     (args.use_thin_pack ? \" thin-pack\" : \"\"),\n-\t\t\t\t     (args.no_progress ? \" no-progress\" : \"\"),\n-\t\t\t\t     (args.include_tag ? \" include-tag\" : \"\"),\n-\t\t\t\t     (prefer_ofs_delta ? \" ofs-delta\" : \"\"));\n-\t\telse\n-\t\t\tpacket_write(fd[1], \"want %s\\n\", sha1_to_hex(remote));\n+\t\tremote_hex = sha1_to_hex(remote);\n+\t\tif (!fetching) {\n+\t\t\tstruct strbuf c = STRBUF_INIT;\n+\t\t\tif (multi_ack)          strbuf_addstr(&c, \" multi_ack\");\n+\t\t\tif (use_sideband == 2)  strbuf_addstr(&c, \" side-band-64k\");\n+\t\t\tif (use_sideband == 1)  strbuf_addstr(&c, \" side-band\");\n+\t\t\tif (args.use_thin_pack) strbuf_addstr(&c, \" thin-pack\");\n+\t\t\tif (args.no_progress)   strbuf_addstr(&c, \" no-progress\");\n+\t\t\tif (args.include_tag)   strbuf_addstr(&c, \" include-tag\");\n+\t\t\tif (prefer_ofs_delta)   strbuf_addstr(&c, \" ofs-delta\");\n+\t\t\tpacket_buf_write(&req_buf, \"want %s%s\\n\", remote_hex, c.buf);\n+\t\t\tstrbuf_release(&c);\n+\t\t} else\n+\t\t\tpacket_buf_write(&req_buf, \"want %s\\n\", remote_hex);\n \t\tfetching++;\n \t}\n+\n+\tif (!fetching) {\n+\t\tstrbuf_release(&req_buf);\n+\t\tpacket_flush(fd[1]);\n+\t\treturn 1;\n+\t}\n+\n \tif (is_repository_shallow())\n-\t\twrite_shallow_commits(fd[1], 1);\n+\t\twrite_shallow_commits(&req_buf, 1);\n \tif (args.depth > 0)\n-\t\tpacket_write(fd[1], \"deepen %d\", args.depth);\n-\tpacket_flush(fd[1]);\n-\tif (!fetching)\n-\t\treturn 1;\n+\t\tpacket_buf_write(&req_buf, \"deepen %d\", args.depth);\n+\tpacket_buf_flush(&req_buf);\n+\n+\tsafe_write(fd[1], req_buf.buf, req_buf.len);\n \n \tif (args.depth > 0) {\n \t\tchar line[1024];\n@@ -296,6 +307,8 @@ done:\n \t\tmulti_ack = 0;\n \t\tflushes++;\n \t}\n+\tstrbuf_release(&req_buf);\n+\n \twhile (flushes || multi_ack) {\n \t\tint ack = get_ack(fd[0], result_sha1);\n \t\tif (ack) {\n@@ -809,6 +822,7 @@ struct ref *fetch_pack(struct fetch_pack_args *my_args,\n \n \tif (args.depth > 0) {\n \t\tstruct cache_time mtime;\n+\t\tstruct strbuf sb = STRBUF_INIT;\n \t\tchar *shallow = git_path(\"shallow\");\n \t\tint fd;\n \n@@ -826,12 +840,14 @@ struct ref *fetch_pack(struct fetch_pack_args *my_args,\n \n \t\tfd = hold_lock_file_for_update(&lock, shallow,\n \t\t\t\t\t       LOCK_DIE_ON_ERROR);\n-\t\tif (!write_shallow_commits(fd, 0)) {\n+\t\tif (!write_shallow_commits(&sb, 0)\n+\t\t || write_in_full(fd, sb.buf, sb.len) != sb.len) {\n \t\t\tunlink_or_warn(shallow);\n \t\t\trollback_lock_file(&lock);\n \t\t} else {\n \t\t\tcommit_lock_file(&lock);\n \t\t}\n+\t\tstrbuf_release(&sb);\n \t}\n \n \treprepare_packed_git();\ndiff --git a/commit.c b/commit.c\nindex fedbd5e..471efb0 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -199,7 +199,7 @@ struct commit_graft *lookup_commit_graft(const unsigned char *sha1)\n \treturn commit_graft[pos];\n }\n \n-int write_shallow_commits(int fd, int use_pack_protocol)\n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n {\n \tint i, count = 0;\n \tfor (i = 0; i < commit_graft_nr; i++)\n@@ -208,12 +208,10 @@ int write_shallow_commits(int fd, int use_pack_protocol)\n \t\t\t\tsha1_to_hex(commit_graft[i]->sha1);\n \t\t\tcount++;\n \t\t\tif (use_pack_protocol)\n-\t\t\t\tpacket_write(fd, \"shallow %s\", hex);\n+\t\t\t\tpacket_buf_write(out, \"shallow %s\", hex);\n \t\t\telse {\n-\t\t\t\tif (write_in_full(fd, hex,  40) != 40)\n-\t\t\t\t\tbreak;\n-\t\t\t\tif (write_str_in_full(fd, \"\\n\") != 1)\n-\t\t\t\t\tbreak;\n+\t\t\t\tstrbuf_addstr(out, hex);\n+\t\t\t\tstrbuf_addch(out, '\\n');\n \t\t\t}\n \t\t}\n \treturn count;\ndiff --git a/commit.h b/commit.h\nindex f4fc5c5..817c75c 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -131,7 +131,7 @@ extern struct commit_list *get_octopus_merge_bases(struct commit_list *in);\n \n extern int register_shallow(const unsigned char *sha1);\n extern int unregister_shallow(const unsigned char *sha1);\n-extern int write_shallow_commits(int fd, int use_pack_protocol);\n+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n extern int is_repository_shallow(void);\n extern struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\tint depth, int shallow_flag, int not_shallow_flag);\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124795","messageId":"1255400715-10508-5-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 04/16] Move \"get_ack()\" back to fetch-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:03Z","receivedAt":"2009-10-13T02:25:03Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"In 41cb7488 Linus moved this function to connect.c for reuse inside\nof the git-clone-pack command.  That was 2005, but in 2006 Junio\nretired git-clone-pack in commit efc7fa53.  Since then the only\ncaller has been fetch-pack.  Since this ACK/NAK exchange is only\nused by the fetch-pack/upload-pack protocol we should keep move\nit back to a private detail of fetch-pack.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n builtin-fetch-pack.c |   21 +++++++++++++++++++++\n cache.h              |    1 -\n connect.c            |   21 ---------------------\n 3 files changed, 21 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex 783c2b0..7c09d46 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -157,6 +157,27 @@ static const unsigned char *get_rev(void)\n \treturn commit->object.sha1;\n }\n \n+static int get_ack(int fd, unsigned char *result_sha1)\n+{\n+\tstatic char line[1000];\n+\tint len = packet_read_line(fd, line, sizeof(line));\n+\n+\tif (!len)\n+\t\tdie(\"git fetch-pack: expected ACK/NAK, got EOF\");\n+\tif (line[len-1] == '\\n')\n+\t\tline[--len] = 0;\n+\tif (!strcmp(line, \"NAK\"))\n+\t\treturn 0;\n+\tif (!prefixcmp(line, \"ACK \")) {\n+\t\tif (!get_sha1_hex(line+4, result_sha1)) {\n+\t\t\tif (strstr(line+45, \"continue\"))\n+\t\t\t\treturn 2;\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\tdie(\"git fetch_pack: expected ACK/NAK, got '%s'\", line);\n+}\n+\n static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t       struct ref *refs)\n {\ndiff --git a/cache.h b/cache.h\nindex a5eeead..4e283be 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -856,7 +856,6 @@ extern struct ref *find_ref_by_name(const struct ref *list, const char *name);\n extern struct child_process *git_connect(int fd[2], const char *url, const char *prog, int flags);\n extern int finish_connect(struct child_process *conn);\n extern int path_match(const char *path, int nr, char **match);\n-extern int get_ack(int fd, unsigned char *result_sha1);\n struct extra_have_objects {\n \tint nr, alloc;\n \tunsigned char (*array)[20];\ndiff --git a/connect.c b/connect.c\nindex 7945e38..839a103 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -107,27 +107,6 @@ int server_supports(const char *feature)\n \t\tstrstr(server_capabilities, feature) != NULL;\n }\n \n-int get_ack(int fd, unsigned char *result_sha1)\n-{\n-\tstatic char line[1000];\n-\tint len = packet_read_line(fd, line, sizeof(line));\n-\n-\tif (!len)\n-\t\tdie(\"git fetch-pack: expected ACK/NAK, got EOF\");\n-\tif (line[len-1] == '\\n')\n-\t\tline[--len] = 0;\n-\tif (!strcmp(line, \"NAK\"))\n-\t\treturn 0;\n-\tif (!prefixcmp(line, \"ACK \")) {\n-\t\tif (!get_sha1_hex(line+4, result_sha1)) {\n-\t\t\tif (strstr(line+45, \"continue\"))\n-\t\t\t\treturn 2;\n-\t\t\treturn 1;\n-\t\t}\n-\t}\n-\tdie(\"git fetch_pack: expected ACK/NAK, got '%s'\", line);\n-}\n-\n int path_match(const char *path, int nr, char **match)\n {\n \tint i;\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124789","messageId":"1255400715-10508-6-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 05/16] Add multi_ack_2 capability to fetch-pack/upload-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:04Z","receivedAt":"2009-10-13T02:25:04Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When multi_ack_2 is enabled the ACK continue messages returned by the\nremote upload-pack are broken out to describe the different states\nwithin the peer.  This permits the client to better understand the\nserver's in-memory state.\n\nThe fetch-pack/upload-pack protocol now looks like:\n\nNAK\n---------------------------------\n  Always sent in response to \"done\" if there was no common base\n  selected from the \"have\" lines (or no have lines were sent).\n\n  * no multi_ack or multi_ack_2:\n\n    Sent when the client has sent a pkt-line flush (\"0000\") and\n    the server has not yet found a common base object.\n\n  * either multi_ack or multi_ack_2:\n\n    Always sent in response to a pkt-line flush.\n\nACK %s\n-----------------------------------\n  * no multi_ack or multi_ack_2:\n\n    Sent in response to \"have\" when the object exists on the remote\n    side and is therefore an object in common between the peers.\n    The argument is the SHA-1 of the common object.\n\n  * either multi_ack or multi_ack_2:\n\n    Sent in response to \"done\" if there are common objects.\n    The argument is the last SHA-1 determined to be common.\n\nACK %s continue\n-----------------------------------\n  * multi_ack only:\n\n    Sent in response to \"have\".\n\n    The remote side wants the client to consider this object as\n    common, and immediately stop transmitting additional \"have\"\n    lines for objects that are reachable from it.  The reason\n    the client should stop is not given, but is one of the two\n    cases below available under multi_ack_2.\n\nACK %s common\n-----------------------------------\n  * multi_ack_2 only:\n\n    Sent in response to \"have\".  Both sides have this object.\n    Like with \"ACK %s continue\" above the client should stop\n    sending have lines reachable for objects from the argument.\n\nACK %s ready\n-----------------------------------\n  * multi_ack_2 only:\n\n    Sent in response to \"have\".\n\n    The client should stop transmitting objects which are reachable\n    from the argument, and send \"done\" soon to get the objects.\n\n    If the remote side has the specified object, it should\n    first send an \"ACK %s common\" message prior to sending\n    \"ACK %s ready\".\n\n    Clients may still submit additional \"have\" lines if there are\n    more side branches for the client to explore that might be added\n    to the common set and reduce the number of objects to transfer.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n builtin-fetch-pack.c |   41 ++++++++++++++++++++++++++++++++---------\n upload-pack.c        |   31 ++++++++++++++++++-------------\n 2 files changed, 50 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex 7c09d46..b68b3eb 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -157,7 +157,15 @@ static const unsigned char *get_rev(void)\n \treturn commit->object.sha1;\n }\n \n-static int get_ack(int fd, unsigned char *result_sha1)\n+enum ack_type {\n+\tNAK = 0,\n+\tACK,\n+\tACK_continue,\n+\tACK_common,\n+\tACK_ready\n+};\n+\n+static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n {\n \tstatic char line[1000];\n \tint len = packet_read_line(fd, line, sizeof(line));\n@@ -167,12 +175,16 @@ static int get_ack(int fd, unsigned char *result_sha1)\n \tif (line[len-1] == '\\n')\n \t\tline[--len] = 0;\n \tif (!strcmp(line, \"NAK\"))\n-\t\treturn 0;\n+\t\treturn NAK;\n \tif (!prefixcmp(line, \"ACK \")) {\n \t\tif (!get_sha1_hex(line+4, result_sha1)) {\n \t\t\tif (strstr(line+45, \"continue\"))\n-\t\t\t\treturn 2;\n-\t\t\treturn 1;\n+\t\t\t\treturn ACK_continue;\n+\t\t\tif (strstr(line+45, \"common\"))\n+\t\t\t\treturn ACK_common;\n+\t\t\tif (strstr(line+45, \"ready\"))\n+\t\t\t\treturn ACK_ready;\n+\t\t\treturn ACK;\n \t\t}\n \t}\n \tdie(\"git fetch_pack: expected ACK/NAK, got '%s'\", line);\n@@ -218,7 +230,8 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\tremote_hex = sha1_to_hex(remote);\n \t\tif (!fetching) {\n \t\t\tstruct strbuf c = STRBUF_INIT;\n-\t\t\tif (multi_ack)          strbuf_addstr(&c, \" multi_ack\");\n+\t\t\tif (multi_ack == 2)     strbuf_addstr(&c, \" multi_ack_2\");\n+\t\t\tif (multi_ack == 1)     strbuf_addstr(&c, \" multi_ack\");\n \t\t\tif (use_sideband == 2)  strbuf_addstr(&c, \" side-band-64k\");\n \t\t\tif (use_sideband == 1)  strbuf_addstr(&c, \" side-band\");\n \t\t\tif (args.use_thin_pack) strbuf_addstr(&c, \" thin-pack\");\n@@ -298,18 +311,23 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t\t\tif (args.verbose && ack)\n \t\t\t\t\tfprintf(stderr, \"got ack %d %s\\n\", ack,\n \t\t\t\t\t\t\tsha1_to_hex(result_sha1));\n-\t\t\t\tif (ack == 1) {\n+\t\t\t\tswitch (ack) {\n+\t\t\t\tcase ACK:\n \t\t\t\t\tflushes = 0;\n \t\t\t\t\tmulti_ack = 0;\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgoto done;\n-\t\t\t\t} else if (ack == 2) {\n+\t\t\t\tcase ACK_common:\n+\t\t\t\tcase ACK_ready:\n+\t\t\t\tcase ACK_continue: {\n \t\t\t\t\tstruct commit *commit =\n \t\t\t\t\t\tlookup_commit(result_sha1);\n \t\t\t\t\tmark_common(commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tin_vain = 0;\n \t\t\t\t\tgot_continue = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t\t}\n \t\t\t\t}\n \t\t\t} while (ack);\n \t\t\tflushes--;\n@@ -336,7 +354,7 @@ done:\n \t\t\tif (args.verbose)\n \t\t\t\tfprintf(stderr, \"got ack (%d) %s\\n\", ack,\n \t\t\t\t\tsha1_to_hex(result_sha1));\n-\t\t\tif (ack == 1)\n+\t\t\tif (ack == ACK)\n \t\t\t\treturn 0;\n \t\t\tmulti_ack = 1;\n \t\t\tcontinue;\n@@ -618,7 +636,12 @@ static struct ref *do_fetch_pack(int fd[2],\n \n \tif (is_repository_shallow() && !server_supports(\"shallow\"))\n \t\tdie(\"Server does not support shallow clients\");\n-\tif (server_supports(\"multi_ack\")) {\n+\tif (server_supports(\"multi_ack_2\")) {\n+\t\tif (args.verbose)\n+\t\t\tfprintf(stderr, \"Server supports multi_ack_2\\n\");\n+\t\tmulti_ack = 2;\n+\t}\n+\telse if (server_supports(\"multi_ack\")) {\n \t\tif (args.verbose)\n \t\t\tfprintf(stderr, \"Server supports multi_ack\\n\");\n \t\tmulti_ack = 1;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 38ddac2..5024b59 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -498,7 +498,7 @@ static int get_common_commits(void)\n {\n \tstatic char line[1000];\n \tunsigned char sha1[20];\n-\tchar hex[41], last_hex[41];\n+\tchar last_hex[41];\n \n \tsave_commit_buffer = 0;\n \n@@ -515,19 +515,22 @@ static int get_common_commits(void)\n \t\tif (!prefixcmp(line, \"have \")) {\n \t\t\tswitch (got_sha1(line+5, sha1)) {\n \t\t\tcase -1: /* they have what we do not */\n-\t\t\t\tif (multi_ack && ok_to_give_up())\n-\t\t\t\t\tpacket_write(1, \"ACK %s continue\\n\",\n-\t\t\t\t\t\t     sha1_to_hex(sha1));\n+\t\t\t\tif (multi_ack && ok_to_give_up()) {\n+\t\t\t\t\tconst char *hex = sha1_to_hex(sha1);\n+\t\t\t\t\tif (multi_ack == 2)\n+\t\t\t\t\t\tpacket_write(1, \"ACK %s ready\\n\", hex);\n+\t\t\t\t\telse\n+\t\t\t\t\t\tpacket_write(1, \"ACK %s continue\\n\", hex);\n+\t\t\t\t}\n \t\t\t\tbreak;\n \t\t\tdefault:\n-\t\t\t\tmemcpy(hex, sha1_to_hex(sha1), 41);\n-\t\t\t\tif (multi_ack) {\n-\t\t\t\t\tconst char *msg = \"ACK %s continue\\n\";\n-\t\t\t\t\tpacket_write(1, msg, hex);\n-\t\t\t\t\tmemcpy(last_hex, hex, 41);\n-\t\t\t\t}\n+\t\t\t\tmemcpy(last_hex, sha1_to_hex(sha1), 41);\n+\t\t\t\tif (multi_ack == 2)\n+\t\t\t\t\tpacket_write(1, \"ACK %s common\\n\", last_hex);\n+\t\t\t\telse if (multi_ack)\n+\t\t\t\t\tpacket_write(1, \"ACK %s continue\\n\", last_hex);\n \t\t\t\telse if (have_obj.nr == 1)\n-\t\t\t\t\tpacket_write(1, \"ACK %s\\n\", hex);\n+\t\t\t\t\tpacket_write(1, \"ACK %s\\n\", last_hex);\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tcontinue;\n@@ -587,7 +590,9 @@ static void receive_needs(void)\n \t\t    get_sha1_hex(line+5, sha1_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n \t\t\t    \"expected to get sha, not '%s'\", line);\n-\t\tif (strstr(line+45, \"multi_ack\"))\n+\t\tif (strstr(line+45, \"multi_ack_2\"))\n+\t\t\tmulti_ack = 2;\n+\t\telse if (strstr(line+45, \"multi_ack\"))\n \t\t\tmulti_ack = 1;\n \t\tif (strstr(line+45, \"thin-pack\"))\n \t\t\tuse_thin_pack = 1;\n@@ -681,7 +686,7 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow no-progress\"\n-\t\t\" include-tag\";\n+\t\t\" include-tag multi_ack_2\";\n \tstruct object *o = parse_object(sha1);\n \n \tif (!o)\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124790","messageId":"1255400715-10508-7-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 06/16] remote-curl: Refactor walker initialization","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:05Z","receivedAt":"2009-10-13T02:25:05Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"We will need the walker, url and remote in other functions as the\ncode grows larger to support smart HTTP.  Extract this out into a\nset of globals we can easily reference once configured.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\nCC: Daniel Barkalow <barkalow@iabervon.org>\n---\n remote-curl.c |   24 ++++++++++++++----------\n 1 files changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex ad6a163..4628ee8 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -4,7 +4,17 @@\n #include \"walker.h\"\n #include \"http.h\"\n \n-static struct ref *get_refs(struct walker *walker, const char *url)\n+static struct remote *remote;\n+static const char *url;\n+static struct walker *walker;\n+\n+static void init_walker(void)\n+{\n+\tif (!walker)\n+\t\twalker = get_http_walker(url, remote);\n+}\n+\n+static struct ref *get_refs(void)\n {\n \tstruct strbuf buffer = STRBUF_INIT;\n \tchar *data, *start, *mid;\n@@ -20,6 +30,7 @@ static struct ref *get_refs(struct walker *walker, const char *url)\n \trefs_url = xmalloc(strlen(url) + 11);\n \tsprintf(refs_url, \"%s/info/refs\", url);\n \n+\tinit_walker();\n \thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n \tswitch (http_ret) {\n \tcase HTTP_OK:\n@@ -77,10 +88,7 @@ static struct ref *get_refs(struct walker *walker, const char *url)\n \n int main(int argc, const char **argv)\n {\n-\tstruct remote *remote;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *url;\n-\tstruct walker *walker = NULL;\n \n \tsetup_git_directory();\n \tif (argc < 2) {\n@@ -101,8 +109,7 @@ int main(int argc, const char **argv)\n \t\t\tbreak;\n \t\tif (!prefixcmp(buf.buf, \"fetch \")) {\n \t\t\tchar *obj = buf.buf + strlen(\"fetch \");\n-\t\t\tif (!walker)\n-\t\t\t\twalker = get_http_walker(url, remote);\n+\t\t\tinit_walker();\n \t\t\twalker->get_all = 1;\n \t\t\twalker->get_tree = 1;\n \t\t\twalker->get_history = 1;\n@@ -113,11 +120,8 @@ int main(int argc, const char **argv)\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n \t\t} else if (!strcmp(buf.buf, \"list\")) {\n-\t\t\tstruct ref *refs;\n+\t\t\tstruct ref *refs = get_refs();\n \t\t\tstruct ref *posn;\n-\t\t\tif (!walker)\n-\t\t\t\twalker = get_http_walker(url, remote);\n-\t\t\trefs = get_refs(walker, url);\n \t\t\tfor (posn = refs; posn; posn = posn->next) {\n \t\t\t\tif (posn->symref)\n \t\t\t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124792","messageId":"1255400715-10508-8-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 07/16] remote-helpers: Fetch more than one ref in a batch","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:06Z","receivedAt":"2009-10-13T02:25:06Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Some network protocols (e.g. native git://) are able to fetch more\nthan one ref at a time and reduce the overall transfer cost by\ncombining the requests into a single exchange.  Instead of feeding\neach fetch request one at a time to the helper, feed all of them\nat once so the helper can decide whether or not it should batch them.\n\nBecause 'fetch' was already released in 1.6.5 we introduce the new\nfetch-multiple capability/command to signal that the helper wants\nto use batch oriented approach to fetching refs.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\nCC: Daniel Barkalow <barkalow@iabervon.org>\n---\n Documentation/git-remote-helpers.txt |   18 ++++++\n remote-curl.c                        |   98 ++++++++++++++++++++++++++++++----\n transport-helper.c                   |   58 +++++++++++++++-----\n 3 files changed, 149 insertions(+), 25 deletions(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex 173ee23..e10ce99 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -43,6 +43,15 @@ Commands are given by the caller on the helper's standard input, one per line.\n +\n Supported if the helper has the \"fetch\" capability.\n \n+'fetch-multiple'::\n+\tFetches multiple objects at once.  The fetch-multiple\n+\tcommand is followed by one or more 'fetch' lines as above,\n+\tand then a blank line to terminate the batch.  Outputs a\n+\tsingle blank line when the entire batch is complete.\n+\tOptionally may output a 'lock <file>' line indicating a\n+\tfile under GIT_DIR/objects/pack which is keeping a pack\n+\tuntil refs can be suitably updated.\n+\n If a fatal error occurs, the program writes the error message to\n stderr and exits. The caller should expect that a suitable error\n message has been printed if the child closes the connection without\n@@ -57,11 +66,20 @@ CAPABILITIES\n 'fetch'::\n \tThis helper supports the 'fetch' command.\n \n+'fetch-multiple'::\n+\tThis helper supports the 'fetch-multiple' command.\n+\n REF LIST ATTRIBUTES\n -------------------\n \n None are defined yet, but the caller must accept any which are supplied.\n \n+FETCH OPTIONS\n+-------------\n+\n+'option verbose'::\n+\tPrint more verbose activity messages to stderr.\n+\n Documentation\n -------------\n Documentation by Daniel Barkalow.\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 4628ee8..34ca4e7 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -86,6 +86,84 @@ static struct ref *get_refs(void)\n \treturn refs;\n }\n \n+static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n+{\n+\tchar **targets = xmalloc(nr_heads * sizeof(char*));\n+\tint ret, i;\n+\n+\tfor (i = 0; i < nr_heads; i++)\n+\t\ttargets[i] = xstrdup(sha1_to_hex(to_fetch[i]->old_sha1));\n+\n+\tinit_walker();\n+\twalker->get_all = 1;\n+\twalker->get_tree = 1;\n+\twalker->get_history = 1;\n+\twalker->get_verbosely = 0;\n+\twalker->get_recover = 0;\n+\tret = walker_fetch(walker, nr_heads, targets, NULL, NULL);\n+\n+\tfor (i = 0; i < nr_heads; i++)\n+\t\tfree(targets[i]);\n+\tfree(targets);\n+\n+\treturn ret ? error(\"Fetch failed.\") : 0;\n+}\n+\n+static void parse_fetch(struct strbuf *buf, int multiple)\n+{\n+\tstruct ref **to_fetch = NULL;\n+\tstruct ref *list_head = NULL;\n+\tstruct ref **list = &list_head;\n+\tint alloc_heads = 0, nr_heads = 0;\n+\n+\tdo {\n+\t\tif (!prefixcmp(buf->buf, \"fetch \")) {\n+\t\t\tchar *p = buf->buf + strlen(\"fetch \");\n+\t\t\tchar *name;\n+\t\t\tstruct ref *ref;\n+\t\t\tunsigned char old_sha1[20];\n+\n+\t\t\tif (strlen(p) < 40 || get_sha1_hex(p, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected sha/ref, got %s'\", p);\n+\t\t\tif (p[40] == ' ')\n+\t\t\t\tname = p + 41;\n+\t\t\telse if (!p[40])\n+\t\t\t\tname = \"\";\n+\t\t\telse\n+\t\t\t\tdie(\"protocol error: expected sha/ref, got %s'\", p);\n+\n+\t\t\tref = alloc_ref(name);\n+\t\t\thashcpy(ref->old_sha1, old_sha1);\n+\n+\t\t\t*list = ref;\n+\t\t\tlist = &ref->next;\n+\n+\t\t\tALLOC_GROW(to_fetch, nr_heads + 1, alloc_heads);\n+\t\t\tto_fetch[nr_heads++] = ref;\n+\n+\t\t\tif (!multiple)\n+\t\t\t\tbreak;\n+\t\t}\n+\t\telse\n+\t\t\tdie(\"http transport does not support %s\", buf->buf);\n+\n+\t\tstrbuf_reset(buf);\n+\t\tif (strbuf_getline(buf, stdin, '\\n') == EOF)\n+\t\t\treturn;\n+\t\tif (!*buf->buf)\n+\t\t\tbreak;\n+\t} while (1);\n+\n+\tif (fetch_dumb(nr_heads, to_fetch))\n+\t\texit(128); /* error already reported */\n+\tfree_refs(list_head);\n+\tfree(to_fetch);\n+\n+\tprintf(\"\\n\");\n+\tfflush(stdout);\n+\tstrbuf_reset(buf);\n+}\n+\n int main(int argc, const char **argv)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -108,17 +186,14 @@ int main(int argc, const char **argv)\n \t\tif (strbuf_getline(&buf, stdin, '\\n') == EOF)\n \t\t\tbreak;\n \t\tif (!prefixcmp(buf.buf, \"fetch \")) {\n-\t\t\tchar *obj = buf.buf + strlen(\"fetch \");\n-\t\t\tinit_walker();\n-\t\t\twalker->get_all = 1;\n-\t\t\twalker->get_tree = 1;\n-\t\t\twalker->get_history = 1;\n-\t\t\twalker->get_verbosely = 0;\n-\t\t\twalker->get_recover = 0;\n-\t\t\tif (walker_fetch(walker, 1, &obj, NULL, NULL))\n-\t\t\t\tdie(\"Fetch failed.\");\n-\t\t\tprintf(\"\\n\");\n-\t\t\tfflush(stdout);\n+\t\t\tparse_fetch(&buf, 0);\n+\n+\t\t} else if (!strcmp(buf.buf, \"fetch-multiple\")) {\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tif (strbuf_getline(&buf, stdin, '\\n') == EOF)\n+\t\t\t\tbreak;\n+\t\t\tparse_fetch(&buf, 1);\n+\n \t\t} else if (!strcmp(buf.buf, \"list\")) {\n \t\t\tstruct ref *refs = get_refs();\n \t\t\tstruct ref *posn;\n@@ -132,6 +207,7 @@ int main(int argc, const char **argv)\n \t\t\tfflush(stdout);\n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n \t\t\tprintf(\"fetch\\n\");\n+\t\t\tprintf(\"fetch-multiple\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n \t\t} else {\ndiff --git a/transport-helper.c b/transport-helper.c\nindex f57e84c..eb66e0c 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -10,7 +10,9 @@ struct helper_data\n {\n \tconst char *name;\n \tstruct child_process *helper;\n-\tunsigned fetch : 1;\n+\tFILE *out;\n+\tunsigned fetch : 1,\n+\t\tfetch_multiple : 1;\n };\n \n static struct child_process *get_helper(struct transport *transport)\n@@ -18,7 +20,6 @@ static struct child_process *get_helper(struct transport *transport)\n \tstruct helper_data *data = transport->data;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct child_process *helper;\n-\tFILE *file;\n \n \tif (data->helper)\n \t\treturn data->helper;\n@@ -39,15 +40,17 @@ static struct child_process *get_helper(struct transport *transport)\n \n \twrite_str_in_full(helper->in, \"capabilities\\n\");\n \n-\tfile = xfdopen(helper->out, \"r\");\n+\tdata->out = xfdopen(helper->out, \"r\");\n \twhile (1) {\n-\t\tif (strbuf_getline(&buf, file, '\\n') == EOF)\n+\t\tif (strbuf_getline(&buf, data->out, '\\n') == EOF)\n \t\t\texit(128); /* child died, message supplied already */\n \n \t\tif (!*buf.buf)\n \t\t\tbreak;\n \t\tif (!strcmp(buf.buf, \"fetch\"))\n \t\t\tdata->fetch = 1;\n+\t\tif (!strcmp(buf.buf, \"fetch-multiple\"))\n+\t\t\tdata->fetch_multiple = 1;\n \t}\n \treturn data->helper;\n }\n@@ -58,6 +61,7 @@ static int disconnect_helper(struct transport *transport)\n \tif (data->helper) {\n \t\twrite_str_in_full(data->helper->in, \"\\n\");\n \t\tclose(data->helper->in);\n+\t\tfclose(data->out);\n \t\tfinish_command(data->helper);\n \t\tfree((char *)data->helper->argv[0]);\n \t\tfree(data->helper->argv);\n@@ -67,14 +71,37 @@ static int disconnect_helper(struct transport *transport)\n \treturn 0;\n }\n \n+static void perform_fetch_command(struct transport *transport,\n+\tstruct strbuf *buf)\n+{\n+\tstruct helper_data *data = transport->data;\n+\tsize_t n = buf->len;\n+\n+\tif (write_in_full(data->helper->in, buf->buf, n) != n)\n+\t\texit(128);\n+\n+\twhile (1) {\n+\t\tstrbuf_reset(buf);\n+\t\tif (strbuf_getline(buf, data->out, '\\n') == EOF)\n+\t\t\texit(128); /* child died, message supplied already */\n+\t\tif (!prefixcmp(buf->buf, \"lock \"))\n+\t\t\ttransport->pack_lockfile = xstrdup(buf->buf + 5);\n+\t\telse if (!buf->len)\n+\t\t\tbreak;\n+\t}\n+\tstrbuf_reset(buf);\n+}\n+\n static int fetch_with_fetch(struct transport *transport,\n \t\t\t    int nr_heads, const struct ref **to_fetch)\n {\n-\tstruct child_process *helper = get_helper(transport);\n-\tFILE *file = xfdopen(helper->out, \"r\");\n+\tstruct helper_data *data = transport->data;\n \tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n \n+\tif (data->fetch_multiple)\n+\t\tstrbuf_addstr(&buf, \"fetch-multiple\\n\");\n+\n \tfor (i = 0; i < nr_heads; i++) {\n \t\tconst struct ref *posn = to_fetch[i];\n \t\tif (posn->status & REF_STATUS_UPTODATE)\n@@ -82,12 +109,16 @@ static int fetch_with_fetch(struct transport *transport,\n \n \t\tstrbuf_addf(&buf, \"fetch %s %s\\n\",\n \t\t\t    sha1_to_hex(posn->old_sha1), posn->name);\n-\t\twrite_in_full(helper->in, buf.buf, buf.len);\n-\t\tstrbuf_reset(&buf);\n+\t\tif (!data->fetch_multiple)\n+\t\t\tperform_fetch_command(transport, &buf);\n+\t}\n \n-\t\tif (strbuf_getline(&buf, file, '\\n') == EOF)\n-\t\t\texit(128); /* child died, message supplied already */\n+\tif (data->fetch_multiple) {\n+\t\tstrbuf_addch(&buf, '\\n');\n+\t\tperform_fetch_command(transport, &buf);\n \t}\n+\n+\tstrbuf_release(&buf);\n \treturn 0;\n }\n \n@@ -105,7 +136,7 @@ static int fetch(struct transport *transport,\n \tif (!count)\n \t\treturn 0;\n \n-\tif (data->fetch)\n+\tif (data->fetch || data->fetch_multiple)\n \t\treturn fetch_with_fetch(transport, nr_heads, to_fetch);\n \n \treturn -1;\n@@ -113,21 +144,20 @@ static int fetch(struct transport *transport,\n \n static struct ref *get_refs_list(struct transport *transport, int for_push)\n {\n+\tstruct helper_data *data = transport->data;\n \tstruct child_process *helper;\n \tstruct ref *ret = NULL;\n \tstruct ref **tail = &ret;\n \tstruct ref *posn;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tFILE *file;\n \n \thelper = get_helper(transport);\n \n \twrite_str_in_full(helper->in, \"list\\n\");\n \n-\tfile = xfdopen(helper->out, \"r\");\n \twhile (1) {\n \t\tchar *eov, *eon;\n-\t\tif (strbuf_getline(&buf, file, '\\n') == EOF)\n+\t\tif (strbuf_getline(&buf, data->out, '\\n') == EOF)\n \t\t\texit(128); /* child died, message supplied already */\n \n \t\tif (!*buf.buf)\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124793","messageId":"1255400715-10508-9-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 08/16] remote-helpers: Support custom transport options","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:07Z","receivedAt":"2009-10-13T02:25:07Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Some transports, like the native pack transport implemented by\nfetch-pack, support useful features like depth or include tags.\nThese should be exposed if the underlying helper knows how to\nuse them and is based upon the same infrastructure.\n\nHelpers must advertise the options they support, any attempt\nto set an unsupported option will cause a failure.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\nCC: Daniel Barkalow <barkalow@iabervon.org>\n---\n Documentation/git-remote-helpers.txt |   20 ++++++++++\n remote-curl.c                        |   16 ++++++-\n transport-helper.c                   |   70 ++++++++++++++++++++++++++++++++++\n 3 files changed, 103 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex e10ce99..334ab30 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -46,6 +46,7 @@ Supported if the helper has the \"fetch\" capability.\n 'fetch-multiple'::\n \tFetches multiple objects at once.  The fetch-multiple\n \tcommand is followed by one or more 'fetch' lines as above,\n+\tzero or more 'option' lines for the supported options,\n \tand then a blank line to terminate the batch.  Outputs a\n \tsingle blank line when the entire batch is complete.\n \tOptionally may output a 'lock <file>' line indicating a\n@@ -69,6 +70,9 @@ CAPABILITIES\n 'fetch-multiple'::\n \tThis helper supports the 'fetch-multiple' command.\n \n+'option' <name>::\n+\tThis helper supports the option <name> under fetch-multiple.\n+\n REF LIST ATTRIBUTES\n -------------------\n \n@@ -76,10 +80,26 @@ None are defined yet, but the caller must accept any which are supplied.\n \n FETCH OPTIONS\n -------------\n+To enable an option the helper must list it in 'capabilities'.\n \n 'option verbose'::\n \tPrint more verbose activity messages to stderr.\n \n+'option uploadpack' <command>::\n+\tThe program to use on the remote side to generate a pack.\n+\n+'option depth' <depth>::\n+\tDeepen the history of a shallow repository.\n+\n+'option keep'::\n+\tKeep the transferred pack(s) with .keep files.\n+\n+'option followtags'::\n+\tAggressively fetch annotated tags if possible.\n+\n+'option thin'::\n+\tTransfer the data as a thin pack if possible.\n+\n Documentation\n -------------\n Documentation by Daniel Barkalow.\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 34ca4e7..e5d9768 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -86,7 +86,12 @@ static struct ref *get_refs(void)\n \treturn refs;\n }\n \n-static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n+struct fetch_args {\n+\tunsigned verbose : 1;\n+};\n+\n+static int fetch_dumb(struct fetch_args *args,\n+\tint nr_heads, struct ref **to_fetch)\n {\n \tchar **targets = xmalloc(nr_heads * sizeof(char*));\n \tint ret, i;\n@@ -98,7 +103,7 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n \twalker->get_all = 1;\n \twalker->get_tree = 1;\n \twalker->get_history = 1;\n-\twalker->get_verbosely = 0;\n+\twalker->get_verbosely = args->verbose;\n \twalker->get_recover = 0;\n \tret = walker_fetch(walker, nr_heads, targets, NULL, NULL);\n \n@@ -115,7 +120,9 @@ static void parse_fetch(struct strbuf *buf, int multiple)\n \tstruct ref *list_head = NULL;\n \tstruct ref **list = &list_head;\n \tint alloc_heads = 0, nr_heads = 0;\n+\tstruct fetch_args args;\n \n+\tmemset(&args, 0, sizeof(args));\n \tdo {\n \t\tif (!prefixcmp(buf->buf, \"fetch \")) {\n \t\t\tchar *p = buf->buf + strlen(\"fetch \");\n@@ -144,6 +151,8 @@ static void parse_fetch(struct strbuf *buf, int multiple)\n \t\t\tif (!multiple)\n \t\t\t\tbreak;\n \t\t}\n+\t\telse if (!strcmp(buf->buf, \"option verbose\"))\n+\t\t\targs.verbose = 1;\n \t\telse\n \t\t\tdie(\"http transport does not support %s\", buf->buf);\n \n@@ -154,7 +163,7 @@ static void parse_fetch(struct strbuf *buf, int multiple)\n \t\t\tbreak;\n \t} while (1);\n \n-\tif (fetch_dumb(nr_heads, to_fetch))\n+\tif (fetch_dumb(&args, nr_heads, to_fetch))\n \t\texit(128); /* error already reported */\n \tfree_refs(list_head);\n \tfree(to_fetch);\n@@ -208,6 +217,7 @@ int main(int argc, const char **argv)\n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n \t\t\tprintf(\"fetch\\n\");\n \t\t\tprintf(\"fetch-multiple\\n\");\n+\t\t\tprintf(\"option verbose\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n \t\t} else {\ndiff --git a/transport-helper.c b/transport-helper.c\nindex eb66e0c..bb6cd1b 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -5,12 +5,15 @@\n #include \"commit.h\"\n #include \"diff.h\"\n #include \"revision.h\"\n+#include \"string-list.h\"\n+\n \n struct helper_data\n {\n \tconst char *name;\n \tstruct child_process *helper;\n \tFILE *out;\n+\tstruct string_list options;\n \tunsigned fetch : 1,\n \t\tfetch_multiple : 1;\n };\n@@ -51,6 +54,11 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\tdata->fetch = 1;\n \t\tif (!strcmp(buf.buf, \"fetch-multiple\"))\n \t\t\tdata->fetch_multiple = 1;\n+\t\tif (!prefixcmp(buf.buf, \"option \")) {\n+\t\t\tconst char *name = buf.buf + strlen(\"option \");\n+\t\t\tif (!string_list_lookup(name, &data->options))\n+\t\t\t\tstring_list_insert(xstrdup(name), &data->options);\n+\t\t}\n \t}\n \treturn data->helper;\n }\n@@ -68,6 +76,65 @@ static int disconnect_helper(struct transport *transport)\n \t\tfree(data->helper);\n \t\tdata->helper = NULL;\n \t}\n+\tstring_list_clear(&data->options, 1);\n+\tfree(data);\n+\treturn 0;\n+}\n+\n+static int save_option(struct transport *transport,\n+\t\t\t  const char *name, const char *value)\n+{\n+\tstruct helper_data *data = transport->data;\n+\tstruct string_list_item *s;\n+\n+\ts = string_list_lookup(name, &data->options);\n+\tif (!s)\n+\t\treturn 1;\n+\tfree(s->util);\n+\ts->util = value ? xstrdup(value) : NULL;\n+\treturn 0;\n+}\n+\n+static int set_helper_option(struct transport *transport,\n+\t\t\t  const char *name, const char *value)\n+{\n+\tstruct helper_data *data = transport->data;\n+\tint is_bool = 0;\n+\n+\tget_helper(transport);\n+\n+\tif (!data->fetch_multiple)\n+\t\treturn 1;\n+\n+\tif (!strcmp(name, TRANS_OPT_THIN))\n+\t\tis_bool = 1;\n+\telse if (!strcmp(name, TRANS_OPT_KEEP))\n+\t\tis_bool = 1;\n+\telse if (!strcmp(name, TRANS_OPT_FOLLOWTAGS))\n+\t\tis_bool = 1;\n+\n+\tif (is_bool)\n+\t\tvalue = value ? \"\" : NULL;\n+\treturn save_option(transport, name, value);\n+}\n+\n+static void standard_options(struct transport *transport)\n+{\n+\tsave_option(transport, \"verbose\", transport->verbose ? \"\" : NULL);\n+}\n+\n+static int print_options(struct string_list_item *s, void *arg)\n+{\n+\tstruct strbuf *buf = arg;\n+\tchar *name = s->string;\n+\tchar *value = s->util;\n+\n+\tif (!value)\n+\t\treturn 0;\n+\telse if (*value)\n+\t\tstrbuf_addf(buf, \"option %s %s\\n\", name, value);\n+\telse\n+\t\tstrbuf_addf(buf, \"option %s\\n\", name);\n \treturn 0;\n }\n \n@@ -114,6 +181,8 @@ static int fetch_with_fetch(struct transport *transport,\n \t}\n \n \tif (data->fetch_multiple) {\n+\t\tstandard_options(transport);\n+\t\tfor_each_string_list(print_options, &data->options, &buf);\n \t\tstrbuf_addch(&buf, '\\n');\n \t\tperform_fetch_command(transport, &buf);\n \t}\n@@ -191,6 +260,7 @@ int transport_helper_init(struct transport *transport, const char *name)\n \tdata->name = name;\n \n \ttransport->data = data;\n+\ttransport->set_option = set_helper_option;\n \ttransport->get_refs_list = get_refs_list;\n \ttransport->fetch = fetch;\n \ttransport->disconnect = disconnect_helper;\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124800","messageId":"1255400715-10508-10-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 09/16] Move WebDAV HTTP push under remote-curl","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:08Z","receivedAt":"2009-10-13T02:25:08Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The remote helper interface now supports the push capability,\nwhich can be used to ask the implementation to push one or more\nspecs to the remote repository.  For remote-curl we implement this\nby calling the existing WebDAV based git-http-push executable.\n\nInternally the helper interface uses the push_refs transport hook\nso that the complexity of the refspec parsing and matching can be\nreused between remote implementations.  When possible however the\nhelper protocol uses source ref name rather than the source SHA-1,\nthereby allowing the helper to access this name if it is useful.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\nCC: Daniel Barkalow <barkalow@iabervon.org>\nCC: Tay Ray Chuan <rctay89@gmail.com>\nCC: Mike Hommey <mh@glandium.org>\n---\n Documentation/git-remote-helpers.txt |   43 ++++++++++-\n http-push.c                          |   43 ++++++++---\n remote-curl.c                        |  101 ++++++++++++++++++++++---\n transport-helper.c                   |  140 +++++++++++++++++++++++++++++++++-\n transport.c                          |   31 --------\n 5 files changed, 302 insertions(+), 56 deletions(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex 334ab30..f8234d0 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -34,6 +34,10 @@ Commands are given by the caller on the helper's standard input, one per line.\n \tvalue of the ref. A space-separated list of attributes follows\n \tthe name; unrecognized attributes are ignored. After the\n \tcomplete list, outputs a blank line.\n++\n+If 'push' is supported this may be called as 'list for-push'\n+to obtain the current refs prior to sending one or more 'push'\n+commands to the helper.\n \n 'fetch' <sha1> <name>::\n \tFetches the given object, writing the necessary objects to the\n@@ -53,6 +57,22 @@ Supported if the helper has the \"fetch\" capability.\n \tfile under GIT_DIR/objects/pack which is keeping a pack\n \tuntil refs can be suitably updated.\n \n+'push' +<src>:<dst>::\n+\tPushes the given <src> commit or branch locally to the\n+\tremote branch described by <dst>.  A batch sequence of\n+\tone or more push commands is terminated with a blank line.\n++\n+Zero or more protocol options may be entered after the last 'push'\n+command, before the batch's terminating blank line.\n++\n+When the push is complete, outputs one or more 'ok <dst>' or\n+'error <dst> <why>?' lines to indicate success or failure of\n+each pushed ref.  The status report output is terminated by\n+a blank line.  The option field <why> may be quoted in a C\n+style string if it contains an LF.\n++\n+Supported if the helper has the \"push\" capability.\n+\n If a fatal error occurs, the program writes the error message to\n stderr and exits. The caller should expect that a suitable error\n message has been printed if the child closes the connection without\n@@ -70,8 +90,12 @@ CAPABILITIES\n 'fetch-multiple'::\n \tThis helper supports the 'fetch-multiple' command.\n \n+'push'::\n+\tThis helper supports the 'push' command.\n+\n 'option' <name>::\n-\tThis helper supports the option <name> under fetch-multiple.\n+\tThis helper supports the option <name> under fetch-multiple\n+\tand push.\n \n REF LIST ATTRIBUTES\n -------------------\n@@ -100,6 +124,23 @@ To enable an option the helper must list it in 'capabilities'.\n 'option thin'::\n \tTransfer the data as a thin pack if possible.\n \n+PUSH OPTIONS\n+------------\n+\n+'option dry-run':\n+\tPretend like the push update will take place, but don't\n+\tactually perform actions which would modify the state of\n+\tthe remote side.\n+\n+'option verbose':\n+\tBe more verbose in progress output to stderr.\n+\n+'option thin'::\n+\tTransfer the data as a thin pack if possible.\n+\n+'option receivepack' <command>::\n+\tThe program to use on the remote side to receive a pack.\n+\n Documentation\n -------------\n Documentation by Daniel Barkalow.\ndiff --git a/http-push.c b/http-push.c\nindex 00e83dc..9010ccc 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -78,6 +78,7 @@ static int push_verbosely;\n static int push_all = MATCH_REFS_NONE;\n static int force_all;\n static int dry_run;\n+static int helper_status;\n \n static struct object_list *objects;\n \n@@ -1813,6 +1814,10 @@ int main(int argc, char **argv)\n \t\t\t\tdry_run = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(arg, \"--helper-status\")) {\n+\t\t\t\thelper_status = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(arg, \"--verbose\")) {\n \t\t\t\tpush_verbosely = 1;\n \t\t\t\thttp_is_verbose = 1;\n@@ -1941,9 +1946,14 @@ int main(int argc, char **argv)\n \n \t\tif (is_null_sha1(ref->peer_ref->new_sha1)) {\n \t\t\tif (delete_remote_branch(ref->name, 1) == -1) {\n-\t\t\t\terror(\"Could not remove %s\", ref->name);\n+\t\t\t\tif (helper_status)\n+\t\t\t\t\tprintf(\"error %s cannot remove\\n\", ref->name);\n+\t\t\t\telse\n+\t\t\t\t\terror(\"Could not remove %s\", ref->name);\n \t\t\t\trc = -4;\n \t\t\t}\n+\t\t\telse if (helper_status)\n+\t\t\t\tprintf(\"ok %s\\n\", ref->name);\n \t\t\tnew_refs++;\n \t\t\tcontinue;\n \t\t}\n@@ -1951,6 +1961,8 @@ int main(int argc, char **argv)\n \t\tif (!hashcmp(ref->old_sha1, ref->peer_ref->new_sha1)) {\n \t\t\tif (push_verbosely || 1)\n \t\t\t\tfprintf(stderr, \"'%s': up-to-date\\n\", ref->name);\n+\t\t\tif (helper_status)\n+\t\t\t\tprintf(\"ok %s up to date\\n\", ref->name);\n \t\t\tcontinue;\n \t\t}\n \n@@ -1968,12 +1980,15 @@ int main(int argc, char **argv)\n \t\t\t\t * commits at the remote end and likely\n \t\t\t\t * we were not up to date to begin with.\n \t\t\t\t */\n-\t\t\t\terror(\"remote '%s' is not an ancestor of\\n\"\n-\t\t\t\t      \"local '%s'.\\n\"\n-\t\t\t\t      \"Maybe you are not up-to-date and \"\n-\t\t\t\t      \"need to pull first?\",\n-\t\t\t\t      ref->name,\n-\t\t\t\t      ref->peer_ref->name);\n+\t\t\t\tif (helper_status)\n+\t\t\t\t\tprintf(\"error %s non-fast forward\\n\", ref->name);\n+\t\t\t\telse\n+\t\t\t\t\terror(\"remote '%s' is not an ancestor of\\n\"\n+\t\t\t\t\t\t  \"local '%s'.\\n\"\n+\t\t\t\t\t\t  \"Maybe you are not up-to-date and \"\n+\t\t\t\t\t\t  \"need to pull first?\",\n+\t\t\t\t\t\t  ref->name,\n+\t\t\t\t\t\t  ref->peer_ref->name);\n \t\t\t\trc = -2;\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -1987,14 +2002,20 @@ int main(int argc, char **argv)\n \t\tif (strcmp(ref->name, ref->peer_ref->name))\n \t\t\tfprintf(stderr, \" using '%s'\", ref->peer_ref->name);\n \t\tfprintf(stderr, \"\\n  from %s\\n  to   %s\\n\", old_hex, new_hex);\n-\t\tif (dry_run)\n+\t\tif (dry_run) {\n+\t\t\tif (helper_status)\n+\t\t\t\tprintf(\"ok %s\\n\", ref->name);\n \t\t\tcontinue;\n+\t\t}\n \n \t\t/* Lock remote branch ref */\n \t\tref_lock = lock_remote(ref->name, LOCK_TIME);\n \t\tif (ref_lock == NULL) {\n-\t\t\tfprintf(stderr, \"Unable to lock remote branch %s\\n\",\n-\t\t\t\tref->name);\n+\t\t\tif (helper_status)\n+\t\t\t\tprintf(\"error %s lock error\\n\", ref->name);\n+\t\t\telse\n+\t\t\t\tfprintf(stderr, \"Unable to lock remote branch %s\\n\",\n+\t\t\t\t\tref->name);\n \t\t\trc = 1;\n \t\t\tcontinue;\n \t\t}\n@@ -2045,6 +2066,8 @@ int main(int argc, char **argv)\n \n \t\tif (!rc)\n \t\t\tfprintf(stderr, \"    done\\n\");\n+\t\tif (helper_status)\n+\t\t\tprintf(\"%s %s\\n\", !rc ? \"ok\" : \"error\", ref->name);\n \t\tunlock_remote(ref_lock);\n \t\tcheck_locks();\n \t}\ndiff --git a/remote-curl.c b/remote-curl.c\nindex e5d9768..000bb52 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -1,8 +1,11 @@\n #include \"cache.h\"\n+#include \"exec_cmd.h\"\n #include \"remote.h\"\n #include \"strbuf.h\"\n #include \"walker.h\"\n #include \"http.h\"\n+#include \"run-command.h\"\n+\n \n static struct remote *remote;\n static const char *url;\n@@ -86,6 +89,20 @@ static struct ref *get_refs(void)\n \treturn refs;\n }\n \n+static void output_refs(struct ref *refs)\n+{\n+\tstruct ref *posn;\n+\tfor (posn = refs; posn; posn = posn->next) {\n+\t\tif (posn->symref)\n+\t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n+\t\telse\n+\t\t\tprintf(\"%s %s\\n\", sha1_to_hex(posn->old_sha1), posn->name);\n+\t}\n+\tprintf(\"\\n\");\n+\tfflush(stdout);\n+\tfree_refs(refs);\n+}\n+\n struct fetch_args {\n \tunsigned verbose : 1;\n };\n@@ -173,10 +190,74 @@ static void parse_fetch(struct strbuf *buf, int multiple)\n \tstrbuf_reset(buf);\n }\n \n+struct push_args {\n+\tunsigned dry_run : 1,\n+\t\tverbose : 1;\n+};\n+\n+static int push_dav(struct push_args *args, int nr_spec, char **specs)\n+{\n+\tconst char **argv = xmalloc((10 + nr_spec) * sizeof(char*));\n+\tint argc = 0, i;\n+\n+\targv[argc++] = \"http-push\";\n+\targv[argc++] = \"--helper-status\";\n+\tif (args->dry_run)\n+\t\targv[argc++] = \"--dry-run\";\n+\tif (args->verbose)\n+\t\targv[argc++] = \"--verbose\";\n+\targv[argc++] = url;\n+\tfor (i = 0; i < nr_spec; i++)\n+\t\targv[argc++] = specs[i];\n+\targv[argc++] = NULL;\n+\n+\tif (run_command_v_opt(argv, RUN_GIT_CMD))\n+\t\tdie(\"git-%s failed\", argv[0]);\n+\tfree(argv);\n+\treturn 0;\n+}\n+\n+static void parse_push(struct strbuf *buf)\n+{\n+\tchar **specs = NULL;\n+\tint alloc_spec = 0, nr_spec = 0, i;\n+\tstruct push_args args;\n+\n+\tmemset(&args, 0, sizeof(args));\n+\tdo {\n+\t\tif (!prefixcmp(buf->buf, \"push \")) {\n+\t\t\tALLOC_GROW(specs, nr_spec + 1, alloc_spec);\n+\t\t\tspecs[nr_spec++] = xstrdup(buf->buf + 5);\n+\n+\t\t} else if (!strcmp(buf->buf, \"option dry-run\"))\n+\t\t\targs.dry_run = 1;\n+\t\telse if (!strcmp(buf->buf, \"option verbose\"))\n+\t\t\targs.verbose = 1;\n+\t\telse\n+\t\t\tdie(\"http transport does not support %s\", buf->buf);\n+\n+\t\tstrbuf_reset(buf);\n+\t\tif (strbuf_getline(buf, stdin, '\\n') == EOF)\n+\t\t\treturn;\n+\t\tif (!*buf->buf)\n+\t\t\tbreak;\n+\t} while (1);\n+\n+\tif (push_dav(&args, nr_spec, specs))\n+\t\texit(128); /* error already reported */\n+\tfor (i = 0; i < nr_spec; i++)\n+\t\tfree(specs[i]);\n+\tfree(specs);\n+\n+\tprintf(\"\\n\");\n+\tfflush(stdout);\n+}\n+\n int main(int argc, const char **argv)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n \n+\tgit_extract_argv0_path(argv[0]);\n \tsetup_git_directory();\n \tif (argc < 2) {\n \t\tfprintf(stderr, \"Remote needed\\n\");\n@@ -204,19 +285,19 @@ int main(int argc, const char **argv)\n \t\t\tparse_fetch(&buf, 1);\n \n \t\t} else if (!strcmp(buf.buf, \"list\")) {\n-\t\t\tstruct ref *refs = get_refs();\n-\t\t\tstruct ref *posn;\n-\t\t\tfor (posn = refs; posn; posn = posn->next) {\n-\t\t\t\tif (posn->symref)\n-\t\t\t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n-\t\t\t\telse\n-\t\t\t\t\tprintf(\"%s %s\\n\", sha1_to_hex(posn->old_sha1), posn->name);\n-\t\t\t}\n-\t\t\tprintf(\"\\n\");\n-\t\t\tfflush(stdout);\n+\t\t\toutput_refs(get_refs());\n+\n+\t\t} else if (!strcmp(buf.buf, \"list for-push\")) {\n+\t\t\toutput_refs(get_refs());\n+\n+\t\t} else if (!prefixcmp(buf.buf, \"push \")) {\n+\t\t\tparse_push(&buf);\n+\n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n \t\t\tprintf(\"fetch\\n\");\n \t\t\tprintf(\"fetch-multiple\\n\");\n+\t\t\tprintf(\"push\\n\");\n+\t\t\tprintf(\"option dry-run\\n\");\n \t\t\tprintf(\"option verbose\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\ndiff --git a/transport-helper.c b/transport-helper.c\nindex bb6cd1b..60fdb16 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1,6 +1,6 @@\n #include \"cache.h\"\n #include \"transport.h\"\n-\n+#include \"quote.h\"\n #include \"run-command.h\"\n #include \"commit.h\"\n #include \"diff.h\"\n@@ -15,7 +15,8 @@ struct helper_data\n \tFILE *out;\n \tstruct string_list options;\n \tunsigned fetch : 1,\n-\t\tfetch_multiple : 1;\n+\t\tfetch_multiple : 1,\n+\t\tpush : 1;\n };\n \n static struct child_process *get_helper(struct transport *transport)\n@@ -54,6 +55,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\tdata->fetch = 1;\n \t\tif (!strcmp(buf.buf, \"fetch-multiple\"))\n \t\t\tdata->fetch_multiple = 1;\n+\t\tif (!strcmp(buf.buf, \"push\"))\n+\t\t\tdata->push = 1;\n \t\tif (!prefixcmp(buf.buf, \"option \")) {\n \t\t\tconst char *name = buf.buf + strlen(\"option \");\n \t\t\tif (!string_list_lookup(name, &data->options))\n@@ -103,7 +106,7 @@ static int set_helper_option(struct transport *transport,\n \n \tget_helper(transport);\n \n-\tif (!data->fetch_multiple)\n+\tif (!data->fetch_multiple && !data->push)\n \t\treturn 1;\n \n \tif (!strcmp(name, TRANS_OPT_THIN))\n@@ -211,6 +214,131 @@ static int fetch(struct transport *transport,\n \treturn -1;\n }\n \n+static int push_refs(struct transport *transport,\n+\t\tstruct ref *remote_refs, int flags)\n+{\n+\tint force_all = flags & TRANSPORT_PUSH_FORCE;\n+\tint mirror = flags & TRANSPORT_PUSH_MIRROR;\n+\tstruct helper_data *data = transport->data;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct child_process *helper;\n+\tstruct ref *ref;\n+\n+\tif (!remote_refs)\n+\t\treturn 0;\n+\n+\thelper = get_helper(transport);\n+\tif (!data->push)\n+\t\treturn 1;\n+\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n+\t\tif (ref->peer_ref)\n+\t\t\thashcpy(ref->new_sha1, ref->peer_ref->new_sha1);\n+\t\telse if (!mirror)\n+\t\t\tcontinue;\n+\n+\t\tref->deletion = is_null_sha1(ref->new_sha1);\n+\t\tif (!ref->deletion &&\n+\t\t\t!hashcmp(ref->old_sha1, ref->new_sha1)) {\n+\t\t\tref->status = REF_STATUS_UPTODATE;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (force_all)\n+\t\t\tref->force = 1;\n+\n+\t\tstrbuf_addstr(&buf, \"push \");\n+\t\tif (!ref->deletion) {\n+\t\t\tif (ref->force)\n+\t\t\t\tstrbuf_addch(&buf, '+');\n+\t\t\tif (ref->peer_ref)\n+\t\t\t\tstrbuf_addstr(&buf, ref->peer_ref->name);\n+\t\t\telse\n+\t\t\t\tstrbuf_addstr(&buf, sha1_to_hex(ref->new_sha1));\n+\t\t}\n+\t\tstrbuf_addch(&buf, ':');\n+\t\tstrbuf_addstr(&buf, ref->name);\n+\t\tstrbuf_addch(&buf, '\\n');\n+\t}\n+\n+\tstandard_options(transport);\n+\tif (flags & TRANSPORT_PUSH_DRY_RUN) {\n+\t\tif (save_option(transport, \"dry-run\", \"\"))\n+\t\t\tdie(\"helper %s does not support dry-run\", data->name);\n+\t}\n+\tif (flags & TRANSPORT_PUSH_VERBOSE)\n+\t\tsave_option(transport, \"verbose\", \"\");\n+\tfor_each_string_list(print_options, &data->options, &buf);\n+\tstrbuf_addch(&buf, '\\n');\n+\n+\tif (write_in_full(helper->in, buf.buf, buf.len) != buf.len)\n+\t\texit(128);\n+\n+\tref = remote_refs;\n+\twhile (1) {\n+\t\tchar *refname, *msg;\n+\t\tint status;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tif (strbuf_getline(&buf, data->out, '\\n') == EOF)\n+\t\t\texit(128); /* child died, message supplied already */\n+\t\tif (!buf.len)\n+\t\t\tbreak;\n+\n+\t\tif (!prefixcmp(buf.buf, \"ok \")) {\n+\t\t\tstatus = REF_STATUS_OK;\n+\t\t\trefname = buf.buf + 3;\n+\t\t} else if (!prefixcmp(buf.buf, \"error \")) {\n+\t\t\tstatus = REF_STATUS_REMOTE_REJECT;\n+\t\t\trefname = buf.buf + 6;\n+\t\t} else\n+\t\t\tdie(\"expected ok/error, helper said '%s'\\n\", buf.buf);\n+\n+\t\tmsg = strchr(refname, ' ');\n+\t\tif (msg) {\n+\t\t\tstruct strbuf msg_buf = STRBUF_INIT;\n+\t\t\tconst char *end;\n+\n+\t\t\t*msg++ = '\\0';\n+\t\t\tif (!unquote_c_style(&msg_buf, msg, &end))\n+\t\t\t\tmsg = strbuf_detach(&msg_buf, NULL);\n+\t\t\telse\n+\t\t\t\tmsg = xstrdup(msg);\n+\t\t\tstrbuf_release(&msg_buf);\n+\n+\t\t\tif (!strcmp(msg, \"no match\")) {\n+\t\t\t\tstatus = REF_STATUS_NONE;\n+\t\t\t\tfree(msg);\n+\t\t\t\tmsg = NULL;\n+\t\t\t}\n+\t\t\telse if (!strcmp(msg, \"up to date\")) {\n+\t\t\t\tstatus = REF_STATUS_UPTODATE;\n+\t\t\t\tfree(msg);\n+\t\t\t\tmsg = NULL;\n+\t\t\t}\n+\t\t\telse if (!strcmp(msg, \"non-fast forward\")) {\n+\t\t\t\tstatus = REF_STATUS_REJECT_NONFASTFORWARD;\n+\t\t\t\tfree(msg);\n+\t\t\t\tmsg = NULL;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref)\n+\t\t\tref = find_ref_by_name(ref, refname);\n+\t\tif (!ref)\n+\t\t\tref = find_ref_by_name(remote_refs, refname);\n+\t\tif (!ref) {\n+\t\t\twarning(\"helper reported unexpected status of %s\", refname);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tref->status = status;\n+\t\tref->remote_status = msg;\n+\t}\n+\tstrbuf_release(&buf);\n+\treturn 0;\n+}\n+\n static struct ref *get_refs_list(struct transport *transport, int for_push)\n {\n \tstruct helper_data *data = transport->data;\n@@ -222,7 +350,10 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)\n \n \thelper = get_helper(transport);\n \n-\twrite_str_in_full(helper->in, \"list\\n\");\n+\tif (data->push && for_push)\n+\t\twrite_str_in_full(helper->in, \"list for-push\\n\");\n+\telse\n+\t\twrite_str_in_full(helper->in, \"list\\n\");\n \n \twhile (1) {\n \t\tchar *eov, *eon;\n@@ -263,6 +394,7 @@ int transport_helper_init(struct transport *transport, const char *name)\n \ttransport->set_option = set_helper_option;\n \ttransport->get_refs_list = get_refs_list;\n \ttransport->fetch = fetch;\n+\ttransport->push_refs = push_refs;\n \ttransport->disconnect = disconnect_helper;\n \treturn 0;\n }\ndiff --git a/transport.c b/transport.c\nindex 644a30a..6d9652d 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -349,35 +349,6 @@ static int rsync_transport_push(struct transport *transport,\n \treturn result;\n }\n \n-#ifndef NO_CURL\n-static int curl_transport_push(struct transport *transport, int refspec_nr, const char **refspec, int flags)\n-{\n-\tconst char **argv;\n-\tint argc;\n-\n-\tif (flags & TRANSPORT_PUSH_MIRROR)\n-\t\treturn error(\"http transport does not support mirror mode\");\n-\n-\targv = xmalloc((refspec_nr + 12) * sizeof(char *));\n-\targv[0] = \"http-push\";\n-\targc = 1;\n-\tif (flags & TRANSPORT_PUSH_ALL)\n-\t\targv[argc++] = \"--all\";\n-\tif (flags & TRANSPORT_PUSH_FORCE)\n-\t\targv[argc++] = \"--force\";\n-\tif (flags & TRANSPORT_PUSH_DRY_RUN)\n-\t\targv[argc++] = \"--dry-run\";\n-\tif (flags & TRANSPORT_PUSH_VERBOSE)\n-\t\targv[argc++] = \"--verbose\";\n-\targv[argc++] = transport->url;\n-\twhile (refspec_nr--)\n-\t\targv[argc++] = *refspec++;\n-\targv[argc] = NULL;\n-\treturn !!run_command_v_opt(argv, RUN_GIT_CMD);\n-}\n-\n-#endif\n-\n struct bundle_transport_data {\n \tint fd;\n \tstruct bundle_header header;\n@@ -826,8 +797,6 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \t\ttransport_helper_init(ret, \"curl\");\n #ifdef NO_CURL\n \t\terror(\"git was compiled without libcurl support.\");\n-#else\n-\t\tret->push = curl_transport_push;\n #endif\n \n \t} else if (is_local(url) && is_file(url)) {\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124794","messageId":"1255400715-10508-11-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 10/16] Git-aware CGI to provide dumb HTTP transport","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:09Z","receivedAt":"2009-10-13T02:25:09Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The git-http-backend CGI can be configured into any Apache server\nusing ScriptAlias, such as with the following configuration:\n\n  LoadModule cgi_module /usr/libexec/apache2/mod_cgi.so\n  LoadModule alias_module /usr/libexec/apache2/mod_alias.so\n  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/\n\nRepositories are accessed via the translated PATH_INFO.\n\nThe CGI is backwards compatible with the dumb client, allowing all\nolder HTTP clients to continue to download repositories which are\nmanaged by the CGI.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n .gitignore     |    1 +\n Makefile       |    1 +\n http-backend.c |  261 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 263 insertions(+), 0 deletions(-)\n create mode 100644 http-backend.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 51a37b1..353d22f 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -55,6 +55,7 @@ git-get-tar-commit-id\n git-grep\n git-hash-object\n git-help\n+git-http-backend\n git-http-fetch\n git-http-push\n git-imap-send\ndiff --git a/Makefile b/Makefile\nindex fea237b..271c290 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -365,6 +365,7 @@ PROGRAMS += git-show-index$X\n PROGRAMS += git-unpack-file$X\n PROGRAMS += git-upload-pack$X\n PROGRAMS += git-var$X\n+PROGRAMS += git-http-backend$X\n \n # List built-in command $C whose implementation cmd_$C() is not in\n # builtin-$C.o but is linked in as part of some other command.\ndiff --git a/http-backend.c b/http-backend.c\nnew file mode 100644\nindex 0000000..39cfd25\n--- /dev/null\n+++ b/http-backend.c\n@@ -0,0 +1,261 @@\n+#include \"cache.h\"\n+#include \"refs.h\"\n+#include \"pkt-line.h\"\n+#include \"object.h\"\n+#include \"tag.h\"\n+#include \"exec_cmd.h\"\n+#include \"run-command.h\"\n+\n+static const char content_type[] = \"Content-Type\";\n+static const char content_length[] = \"Content-Length\";\n+\n+static char buffer[1024];\n+\n+static const char *http_date(unsigned long time)\n+{\n+\treturn show_date(time, 0, DATE_RFC2822);\n+}\n+\n+static void format_write(const char *fmt, ...)\n+{\n+\tva_list args;\n+\tunsigned n;\n+\n+\tva_start(args, fmt);\n+\tn = vsnprintf(buffer, sizeof(buffer), fmt, args);\n+\tva_end(args);\n+\tif (n >= sizeof(buffer))\n+\t\tdie(\"protocol error: impossibly long line\");\n+\n+\tsafe_write(1, buffer, n);\n+}\n+\n+static void write_status(unsigned code, const char *msg)\n+{\n+\tformat_write(\"Status: %u %s\\r\\n\", code, msg);\n+}\n+\n+static void write_header(const char *name, const char *value)\n+{\n+\tformat_write(\"%s: %s\\r\\n\", name, value);\n+}\n+\n+static void end_headers(void)\n+{\n+\tsafe_write(1, \"\\r\\n\", 2);\n+}\n+\n+static void write_nocache(void)\n+{\n+\twrite_header(\"Expires\", \"Fri, 01 Jan 1980 00:00:00 GMT\");\n+\twrite_header(\"Pragma\", \"no-cache\");\n+\twrite_header(\"Cache-Control\", \"no-cache, max-age=0, must-revalidate\");\n+}\n+\n+static void write_cache_forever(void)\n+{\n+\tunsigned long now = time(NULL);\n+\twrite_header(\"Date\", http_date(now));\n+\twrite_header(\"Expires\", http_date(now + 31536000));\n+\twrite_header(\"Cache-Control\", \"public, max-age=31536000\");\n+}\n+\n+static NORETURN void not_found(const char *err, ...)\n+{\n+\tva_list params;\n+\n+\twrite_status(404, \"Not Found\");\n+\twrite_nocache();\n+\tend_headers();\n+\n+\tva_start(params, err);\n+\tif (err && *err) {\n+\t\tvsnprintf(buffer, sizeof(buffer), err, params);\n+\t\tfprintf(stderr, \"%s\\n\", buffer);\n+\t}\n+\tva_end(params);\n+\texit(0);\n+}\n+\n+static void write_file(const char *the_type, const char *name)\n+{\n+\tconst char *p = git_path(\"%s\", name);\n+\tint fd;\n+\tstruct stat sb;\n+\tuintmax_t remaining;\n+\n+\tfd = open(p, O_RDONLY);\n+\tif (fd < 0)\n+\t\tnot_found(\"Cannot open '%s': %s\", p, strerror(errno));\n+\tif (fstat(fd, &sb) < 0)\n+\t\tdie_errno(\"Cannot stat '%s'\", p);\n+\tremaining = (uintmax_t)sb.st_size;\n+\n+\twrite_header(content_type, the_type);\n+\twrite_header(\"Last-Modified\", http_date(sb.st_mtime));\n+\tformat_write(\"Content-Length: %\" PRIuMAX \"\\r\\n\", remaining);\n+\tend_headers();\n+\n+\twhile (remaining) {\n+\t\tssize_t n = xread(fd, buffer, sizeof(buffer));\n+\t\tif (n < 0)\n+\t\t\tdie_errno(\"Cannot read '%s'\", p);\n+\t\tn = safe_write(1, buffer, n);\n+\t\tif (n <= 0)\n+\t\t\tbreak;\n+\t}\n+\tclose(fd);\n+}\n+\n+static void get_text_file(char *name)\n+{\n+\twrite_nocache();\n+\twrite_file(\"text/plain; charset=utf-8\", name);\n+}\n+\n+static void get_loose_object(char *name)\n+{\n+\twrite_cache_forever();\n+\twrite_file(\"application/x-git-loose-object\", name);\n+}\n+\n+static void get_pack_file(char *name)\n+{\n+\twrite_cache_forever();\n+\twrite_file(\"application/x-git-packed-objects\", name);\n+}\n+\n+static void get_idx_file(char *name)\n+{\n+\twrite_cache_forever();\n+\twrite_file(\"application/x-git-packed-objects-toc\", name);\n+}\n+\n+static int show_text_ref(const char *name, const unsigned char *sha1,\n+\tint flag, void *cb_data)\n+{\n+\tstruct object *o = parse_object(sha1);\n+\tif (!o)\n+\t\treturn 0;\n+\n+\tformat_write(\"%s\\t%s\\n\", sha1_to_hex(sha1), name);\n+\tif (o->type == OBJ_TAG) {\n+\t\to = deref_tag(o, name, 0);\n+\t\tif (!o)\n+\t\t\treturn 0;\n+\t\tformat_write(\"%s\\t%s^{}\\n\", sha1_to_hex(o->sha1), name);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void get_info_refs(char *arg)\n+{\n+\twrite_nocache();\n+\twrite_header(content_type, \"text/plain; charset=utf-8\");\n+\tend_headers();\n+\n+\tfor_each_ref(show_text_ref, NULL);\n+}\n+\n+static void get_info_packs(char *arg)\n+{\n+\tsize_t objdirlen = strlen(get_object_directory());\n+\tstruct packed_git *p;\n+\n+\twrite_nocache();\n+\twrite_header(content_type, \"text/plain; charset=utf-8\");\n+\tend_headers();\n+\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif (!p->pack_local)\n+\t\t\tcontinue;\n+\t\tformat_write(\"P %s\\n\", p->pack_name + objdirlen + 6);\n+\t}\n+\tsafe_write(1, \"\\n\", 1);\n+}\n+\n+static NORETURN void die_webcgi(const char *err, va_list params)\n+{\n+\twrite_status(500, \"Internal Server Error\");\n+\twrite_nocache();\n+\tend_headers();\n+\n+\tvsnprintf(buffer, sizeof(buffer), err, params);\n+\tfprintf(stderr, \"fatal: %s\\n\", buffer);\n+\texit(0);\n+}\n+\n+static struct service_cmd {\n+\tconst char *method;\n+\tconst char *pattern;\n+\tvoid (*imp)(char *);\n+} services[] = {\n+\t{\"GET\", \"/HEAD$\", get_text_file},\n+\t{\"GET\", \"/info/refs$\", get_info_refs},\n+\t{\"GET\", \"/objects/info/packs$\", get_info_packs},\n+\t{\"GET\", \"/objects/info/[^/]*$\", get_text_file},\n+\t{\"GET\", \"/objects/[0-9a-f]{2}/[0-9a-f]{38}$\", get_loose_object},\n+\t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.pack$\", get_pack_file},\n+\t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.idx$\", get_idx_file}\n+};\n+\n+int main(int argc, char **argv)\n+{\n+\tchar *dir = getenv(\"PATH_TRANSLATED\");\n+\tchar *input_method = getenv(\"REQUEST_METHOD\");\n+\tstruct service_cmd *cmd = NULL;\n+\tchar *cmd_arg = NULL;\n+\tint i;\n+\n+\tset_die_routine(die_webcgi);\n+\n+\tif (!dir)\n+\t\tdie(\"No PATH_TRANSLATED from server\");\n+\tif (!input_method)\n+\t\tdie(\"No REQUEST_METHOD from server\");\n+\tif (!strcmp(input_method, \"HEAD\"))\n+\t\tinput_method = \"GET\";\n+\n+\tfor (i = 0; i < ARRAY_SIZE(services); i++) {\n+\t\tstruct service_cmd *c = &services[i];\n+\t\tregex_t re;\n+\t\tregmatch_t out[1];\n+\n+\t\tif (regcomp(&re, c->pattern, REG_EXTENDED))\n+\t\t\tdie(\"Bogus regex in service table: %s\", c->pattern);\n+\t\tif (!regexec(&re, dir, 1, out, 0)) {\n+\t\t\tsize_t n = out[0].rm_eo - out[0].rm_so;\n+\n+\t\t\tif (strcmp(input_method, c->method)) {\n+\t\t\t\tconst char *proto = getenv(\"SERVER_PROTOCOL\");\n+\t\t\t\tif (proto && !strcmp(proto, \"HTTP/1.1\"))\n+\t\t\t\t\twrite_status(405, \"Method Not Allowed\");\n+\t\t\t\telse\n+\t\t\t\t\twrite_status(400, \"Bad Request\");\n+\t\t\t\twrite_nocache();\n+\t\t\t\tend_headers();\n+\t\t\t\treturn 0;\n+\t\t\t}\n+\n+\t\t\tcmd = c;\n+\t\t\tcmd_arg = xmalloc(n);\n+\t\t\tstrncpy(cmd_arg, dir + out[0].rm_so + 1, n);\n+\t\t\tcmd_arg[n] = '\\0';\n+\t\t\tdir[out[0].rm_so] = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tregfree(&re);\n+\t}\n+\n+\tif (!cmd)\n+\t\tnot_found(\"Request not supported: '%s'\", dir);\n+\n+\tsetup_path();\n+\tif (!enter_repo(dir, 0))\n+\t\tnot_found(\"Not a git repository: '%s'\", dir);\n+\n+\tcmd->imp(cmd_arg);\n+\treturn 0;\n+}\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124797","messageId":"1255400715-10508-12-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 11/16] Add one shot RPC options to upload-pack, receive-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:10Z","receivedAt":"2009-10-13T02:25:10Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When --one-shot-rpc is passed as a command line parameter to\nupload-pack or receive-pack the programs now assume they may\nperform only a single read-write cycle with stdin and stdout.\nThis fits with the HTTP POST request processing model where a\nprogram may read the request, write a response, and must exit.\n\nWhen --advertise-refs is passed as a command line parameter only\nthe initial ref advertisement is output, and the program exits\nimmediately.  This fits with the HTTP GET request model, where\nno request content is received but a response must be produced.\n\nHTTP headers and/or environment are not processed here, but\ninstead are assumed to be handled by the program invoking\neither service backend.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n builtin-receive-pack.c |   26 ++++++++++++++++++++------\n upload-pack.c          |   40 ++++++++++++++++++++++++++++++++++++----\n 2 files changed, 56 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin-receive-pack.c b/builtin-receive-pack.c\nindex b771fe9..9e8f36f 100644\n--- a/builtin-receive-pack.c\n+++ b/builtin-receive-pack.c\n@@ -615,6 +615,8 @@ static void add_alternate_refs(void)\n \n int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n {\n+\tint advertise_refs = 0;\n+\tint one_shot_rpc = 0;\n \tint i;\n \tchar *dir = NULL;\n \n@@ -623,7 +625,15 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tconst char *arg = *argv++;\n \n \t\tif (*arg == '-') {\n-\t\t\t/* Do flag handling here */\n+\t\t\tif (!strcmp(arg, \"--advertise-refs\")) {\n+\t\t\t\tadvertise_refs = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!strcmp(arg, \"--one-shot-rpc\")) {\n+\t\t\t\tone_shot_rpc = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tusage(receive_pack_usage);\n \t\t}\n \t\tif (dir)\n@@ -652,12 +662,16 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\t\" report-status delete-refs ofs-delta \" :\n \t\t\" report-status delete-refs \";\n \n-\tadd_alternate_refs();\n-\twrite_head_info();\n-\tclear_extra_refs();\n+\tif (advertise_refs || !one_shot_rpc) {\n+\t\tadd_alternate_refs();\n+\t\twrite_head_info();\n+\t\tclear_extra_refs();\n \n-\t/* EOF */\n-\tpacket_flush(1);\n+\t\t/* EOF */\n+\t\tpacket_flush(1);\n+\t}\n+\tif (advertise_refs)\n+\t\treturn 0;\n \n \tread_head_info();\n \tif (commands) {\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 5024b59..71318d7 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -39,6 +39,8 @@ static unsigned int timeout;\n  */\n static int use_sideband;\n static int debug_fd;\n+static int advertise_refs;\n+static int one_shot_rpc;\n \n static void reset_timeout(void)\n {\n@@ -509,6 +511,8 @@ static int get_common_commits(void)\n \t\tif (!len) {\n \t\t\tif (have_obj.nr == 0 || multi_ack)\n \t\t\t\tpacket_write(1, \"NAK\\n\");\n+\t\t\tif (one_shot_rpc)\n+\t\t\t\texit(0);\n \t\t\tcontinue;\n \t\t}\n \t\tstrip(line, len);\n@@ -710,12 +714,32 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n \treturn 0;\n }\n \n+static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)\n+{\n+\tstruct object *o = parse_object(sha1);\n+\tif (!o)\n+\t\tdie(\"git upload-pack: cannot find object %s:\", sha1_to_hex(sha1));\n+\tif (!(o->flags & OUR_REF)) {\n+\t\to->flags |= OUR_REF;\n+\t\tnr_our_refs++;\n+\t}\n+\treturn 0;\n+}\n+\n static void upload_pack(void)\n {\n-\treset_timeout();\n-\thead_ref(send_ref, NULL);\n-\tfor_each_ref(send_ref, NULL);\n-\tpacket_flush(1);\n+\tif (advertise_refs || !one_shot_rpc) {\n+\t\treset_timeout();\n+\t\thead_ref(send_ref, NULL);\n+\t\tfor_each_ref(send_ref, NULL);\n+\t\tpacket_flush(1);\n+\t} else {\n+\t\thead_ref(mark_our_ref, NULL);\n+\t\tfor_each_ref(mark_our_ref, NULL);\n+\t}\n+\tif (advertise_refs)\n+\t\treturn;\n+\n \treceive_needs();\n \tif (want_obj.nr) {\n \t\tget_common_commits();\n@@ -737,6 +761,14 @@ int main(int argc, char **argv)\n \n \t\tif (arg[0] != '-')\n \t\t\tbreak;\n+\t\tif (!strcmp(arg, \"--advertise-refs\")) {\n+\t\t\tadvertise_refs = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--one-shot-rpc\")) {\n+\t\t\tone_shot_rpc = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--strict\")) {\n \t\t\tstrict = 1;\n \t\t\tcontinue;\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124801","messageId":"1255400715-10508-13-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 12/16] Smart fetch and push over HTTP: server side","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:11Z","receivedAt":"2009-10-13T02:25:11Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Requests for $GIT_URL/git-receive-pack and $GIT_URL/git-upload-pack\nare forwarded to the corresponding backend process by directly\nexecuting it and leaving stdin and stdout connected to the invoking\nweb server.  Prior to starting the backend process the HTTP response\nheaders are sent, thereby freeing the backend from needing to know\nabout the HTTP protocol.\n\nRequests that are encoded with Content-Encoding: gzip are\nautomatically inflated before being streamed into the backend.\nThis is primarily useful for the git-upload-pack backend, which\nreceives highly repetitive text data from clients that easily\ncompresses to 50% of its original size.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\n---\n http-backend.c |  192 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 files changed, 191 insertions(+), 1 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex 39cfd25..adb3256 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -77,6 +77,152 @@ static NORETURN void not_found(const char *err, ...)\n \texit(0);\n }\n \n+static NORETURN void forbidden(const char *err, ...)\n+{\n+\tva_list params;\n+\n+\twrite_status(403, \"Forbidden\");\n+\twrite_nocache();\n+\tend_headers();\n+\n+\tva_start(params, err);\n+\tif (err && *err) {\n+\t\tvsnprintf(buffer, sizeof(buffer), err, params);\n+\t\tfprintf(stderr, \"%s\\n\", buffer);\n+\t}\n+\tva_end(params);\n+\texit(0);\n+}\n+\n+struct http_service {\n+\tconst char *name;\n+\tconst char *config_name;\n+\tint enabled;\n+};\n+static struct http_service *service;\n+\n+static struct http_service http_service[] = {\n+\t{ \"upload-pack\", \"uploadpack\", 1 },\n+\t{ \"receive-pack\", \"receivepack\", 0 },\n+};\n+\n+static int http_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!prefixcmp(var, \"http.\") &&\n+\t    !strcmp(var + 5, service->config_name)) {\n+\t\tservice->enabled = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\t/* we are not interested in parsing any other configuration here */\n+\treturn 0;\n+}\n+\n+static void select_service(const char *name)\n+{\n+\tint i;\n+\n+\tif (prefixcmp(name, \"git-\"))\n+\t\tforbidden(\"Unsupported service: '%s'\", name);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(http_service); i++) {\n+\t\tservice = &http_service[i];\n+\t\tif (!strcmp(service->name, name + 4)) {\n+\t\t\tgit_config(http_config, NULL);\n+\t\t\tif (!service->enabled)\n+\t\t\t\tforbidden(\"Service not enabled: '%s'\", name);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\tforbidden(\"Unsupported service: '%s'\", name);\n+}\n+\n+static void inflate_request(const char *prog_name, int out)\n+{\n+\tz_stream stream;\n+\tunsigned char in_buf[8192];\n+\tunsigned char out_buf[8192];\n+\tunsigned long cnt = 0;\n+\tint ret;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tret = inflateInit2(&stream, (15 + 16));\n+\tif (ret != Z_OK)\n+\t\tdie(\"cannot start zlib inflater, zlib err %d\", ret);\n+\n+\twhile (1) {\n+\t\tssize_t n = xread(0, in_buf, sizeof(in_buf));\n+\t\tif (n <= 0)\n+\t\t\tdie(\"request ended in the middle of the gzip stream\");\n+\n+\t\tstream.next_in = in_buf;\n+\t\tstream.avail_in = n;\n+\n+\t\twhile (0 < stream.avail_in) {\n+\t\t\tint ret;\n+\n+\t\t\tstream.next_out = out_buf;\n+\t\t\tstream.avail_out = sizeof(out_buf);\n+\n+\t\t\tret = inflate(&stream, Z_NO_FLUSH);\n+\t\t\tif (ret != Z_OK && ret != Z_STREAM_END)\n+\t\t\t\tdie(\"zlib error inflating request, result %d\", ret);\n+\n+\t\t\tn = stream.total_out - cnt;\n+\t\t\tif (write_in_full(out, out_buf, n) != n)\n+\t\t\t\tdie(\"%s aborted reading request\", prog_name);\n+\t\t\tcnt += n;\n+\n+\t\t\tif (ret == Z_STREAM_END)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tinflateEnd(&stream);\n+\tclose(out);\n+}\n+\n+static void run_service(const char **argv)\n+{\n+\tconst char *encoding = getenv(\"HTTP_CONTENT_ENCODING\");\n+\tint use_gzip = 0;\n+\tstruct child_process cld;\n+\n+\tif (encoding && !strcmp(encoding, \"gzip\"))\n+\t\tuse_gzip = 1;\n+\telse if (encoding && !strcmp(encoding, \"x-gzip\"))\n+\t\tuse_gzip = 1;\n+\n+\tmemset(&cld, 0, sizeof(cld));\n+\tcld.argv = argv;\n+\tif (use_gzip)\n+\t\tcld.in = -1;\n+\tcld.git_cmd = 1;\n+\tif (start_command(&cld))\n+\t\tdie_errno(\"Cannot start %s\", argv[0]);\n+\n+\tclose(1);\n+\tif (use_gzip)\n+\t\tinflate_request(argv[0], cld.in);\n+\telse\n+\t\tclose(0);\n+\n+\tif (finish_command(&cld))\n+\t\tdie(\"%s terminated with error\", argv[0]);\n+}\n+\n+static void require_content_type(const char *need_type)\n+{\n+\tconst char *input_type = getenv(\"CONTENT_TYPE\");\n+\tif (!input_type || strcmp(input_type, need_type)) {\n+\t\twrite_status(415, \"Unsupported Media Type\");\n+\t\twrite_nocache();\n+\t\tend_headers();\n+\t\texit(0);\n+\t}\n+}\n+\n static void write_file(const char *the_type, const char *name)\n {\n \tconst char *p = git_path(\"%s\", name);\n@@ -151,6 +297,25 @@ static int show_text_ref(const char *name, const unsigned char *sha1,\n \n static void get_info_refs(char *arg)\n {\n+\tchar *query = getenv(\"QUERY_STRING\");\n+\n+\tif (query && !prefixcmp(query, \"service=\")) {\n+\t\tconst char *argv[] = {NULL /* service name */,\n+\t\t\t\"--one-shot-rpc\", \"--advertise-refs\",\n+\t\t\t\".\", NULL};\n+\n+\t\tselect_service(query + 8);\n+\n+\t\twrite_nocache();\n+\t\tformat_write(\"%s: application/x-git-%s-advertisement\\r\\n\",\n+\t\t\tcontent_type, service->name);\n+\t\tend_headers();\n+\t\tpacket_write(1, \"# service=git-%s\\n\", service->name);\n+\n+\t\targv[0] = service->name;\n+\t\trun_service(argv);\n+\t}\n+\n \twrite_nocache();\n \twrite_header(content_type, \"text/plain; charset=utf-8\");\n \tend_headers();\n@@ -176,6 +341,28 @@ static void get_info_packs(char *arg)\n \tsafe_write(1, \"\\n\", 1);\n }\n \n+static void post_to_service(char *service_name)\n+{\n+\tconst char *argv[] = {NULL, \"--one-shot-rpc\", \".\", NULL};\n+\tunsigned n;\n+\n+\tselect_service(service_name);\n+\n+\tn = snprintf(buffer, sizeof(buffer),\n+\t\t\"application/x-git-%s-request\", service->name);\n+\tif (n >= sizeof(buffer))\n+\t\tdie(\"impossibly long service name\");\n+\trequire_content_type(buffer);\n+\n+\twrite_nocache();\n+\tformat_write(\"%s: application/x-git-%s-result\\r\\n\",\n+\t\tcontent_type, service->name);\n+\tend_headers();\n+\n+\targv[0] = service->name;\n+\trun_service(argv);\n+}\n+\n static NORETURN void die_webcgi(const char *err, va_list params)\n {\n \twrite_status(500, \"Internal Server Error\");\n@@ -198,7 +385,10 @@ static struct service_cmd {\n \t{\"GET\", \"/objects/info/[^/]*$\", get_text_file},\n \t{\"GET\", \"/objects/[0-9a-f]{2}/[0-9a-f]{38}$\", get_loose_object},\n \t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.pack$\", get_pack_file},\n-\t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.idx$\", get_idx_file}\n+\t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.idx$\", get_idx_file},\n+\n+\t{\"POST\", \"/git-upload-pack$\", post_to_service},\n+\t{\"POST\", \"/git-receive-pack$\", post_to_service}\n };\n \n int main(int argc, char **argv)\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124798","messageId":"1255400715-10508-14-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 13/16] Discover refs via smart HTTP server when available","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:12Z","receivedAt":"2009-10-13T02:25:12Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Instead of loading the cached info/refs, try to use the smart HTTP\nversion when the server supports it.  Since the smart variant is\nactually the pkt-line stream from the start of either upload-pack\nor receive-pack we need to parse these through get_remote_heads,\nwhich requires a background thread to feed its pipe.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\nCC: Daniel Barkalow <barkalow@iabervon.org>\n---\n remote-curl.c |  135 +++++++++++++++++++++++++++++++++++++++++++++++++--------\n 1 files changed, 116 insertions(+), 19 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 000bb52..42fd06c 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -4,6 +4,7 @@\n #include \"strbuf.h\"\n #include \"walker.h\"\n #include \"http.h\"\n+#include \"pkt-line.h\"\n #include \"run-command.h\"\n \n \n@@ -17,24 +18,41 @@ static void init_walker(void)\n \t\twalker = get_http_walker(url, remote);\n }\n \n-static struct ref *get_refs(void)\n+struct discovery {\n+\tconst char *service;\n+\tchar *buf_alloc;\n+\tchar *buf;\n+\tsize_t len;\n+\tunsigned proto_git : 1;\n+};\n+static struct discovery *last_discovery;\n+\n+static void free_discovery(struct discovery *d)\n {\n-\tstruct strbuf buffer = STRBUF_INIT;\n-\tchar *data, *start, *mid;\n-\tchar *ref_name;\n+\tif (d) {\n+\t\tif (d == last_discovery)\n+\t\t\tlast_discovery = NULL;\n+\t\tfree(d->buf_alloc);\n+\t\tfree(d);\n+\t}\n+}\n+\n+static struct discovery* discover_refs(const char *service)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct discovery *last = last_discovery;\n \tchar *refs_url;\n-\tint i = 0;\n \tint http_ret;\n \n-\tstruct ref *refs = NULL;\n-\tstruct ref *ref = NULL;\n-\tstruct ref *last_ref = NULL;\n+\tif (last && !strcmp(service, last->service))\n+\t\treturn last;\n+\tfree_discovery(last);\n \n-\trefs_url = xmalloc(strlen(url) + 11);\n-\tsprintf(refs_url, \"%s/info/refs\", url);\n+\tstrbuf_addf(&buf, \"%s/info/refs?service=%s\", url, service);\n+\trefs_url = strbuf_detach(&buf, NULL);\n \n \tinit_walker();\n-\thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n+\thttp_ret = http_get_strbuf(refs_url, &buf, HTTP_NO_CACHE);\n \tswitch (http_ret) {\n \tcase HTTP_OK:\n \t\tbreak;\n@@ -46,10 +64,78 @@ static struct ref *get_refs(void)\n \t\tdie(\"HTTP request failed\");\n \t}\n \n-\tdata = buffer.buf;\n+\tlast= xcalloc(1, sizeof(*last_discovery));\n+\tlast->service = service;\n+\tlast->buf_alloc = strbuf_detach(&buf, &last->len);\n+\tlast->buf = last->buf_alloc;\n+\n+\tif (5 <= last->len && last->buf[4] == '#') {\n+\t\t/* smart HTTP response; validate that the service\n+\t\t * pkt-line matches our request.\n+\t\t */\n+\t\tstruct strbuf exp = STRBUF_INIT;\n+\n+\t\tif (packet_get_line(&buf, &last->buf, &last->len) <= 0)\n+\t\t\tdie(\"%s has invalid packet header\", refs_url);\n+\t\tif (buf.len && buf.buf[buf.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&buf, buf.len - 1);\n+\n+\t\tstrbuf_addf(&exp, \"# service=%s\", service);\n+\t\tif (strbuf_cmp(&exp, &buf))\n+\t\t\tdie(\"invalid server response; got '%s'\", buf.buf);\n+\t\tstrbuf_release(&exp);\n+\n+\t\tlast->proto_git = 1;\n+\t}\n+\n+\tfree(refs_url);\n+\tstrbuf_release(&buf);\n+\tlast_discovery = last;\n+\treturn last;\n+}\n+\n+static int write_discovery(int fd, void *data)\n+{\n+\tstruct discovery *heads = data;\n+\tint err = 0;\n+\tif (write_in_full(fd, heads->buf, heads->len) != heads->len)\n+\t\terr = 1;\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+static struct ref *parse_git_refs(struct discovery *heads)\n+{\n+\tstruct ref *list = NULL;\n+\tstruct async async;\n+\n+\tmemset(&async, 0, sizeof(async));\n+\tasync.proc = write_discovery;\n+\tasync.data = heads;\n+\n+\tif (start_async(&async))\n+\t\tdie(\"cannot start thread to parse advertised refs\");\n+\tget_remote_heads(async.out, &list, 0, NULL, 0, NULL);\n+\tclose(async.out);\n+\tif (finish_async(&async))\n+\t\tdie(\"ref parsing thread failed\");\n+\treturn list;\n+}\n+\n+static struct ref *parse_info_refs(struct discovery *heads)\n+{\n+\tchar *data, *start, *mid;\n+\tchar *ref_name;\n+\tint i = 0;\n+\n+\tstruct ref *refs = NULL;\n+\tstruct ref *ref = NULL;\n+\tstruct ref *last_ref = NULL;\n+\n+\tdata = heads->buf;\n \tstart = NULL;\n \tmid = data;\n-\twhile (i < buffer.len) {\n+\twhile (i < heads->len) {\n \t\tif (!start) {\n \t\t\tstart = &data[i];\n \t\t}\n@@ -73,8 +159,7 @@ static struct ref *get_refs(void)\n \t\ti++;\n \t}\n \n-\tstrbuf_release(&buffer);\n-\n+\tinit_walker();\n \tref = alloc_ref(\"HEAD\");\n \tif (!walker->fetch_ref(walker, ref) &&\n \t    !resolve_remote_symref(ref, refs)) {\n@@ -84,11 +169,23 @@ static struct ref *get_refs(void)\n \t\tfree(ref);\n \t}\n \n-\tstrbuf_release(&buffer);\n-\tfree(refs_url);\n \treturn refs;\n }\n \n+static struct ref *get_refs(int for_push)\n+{\n+\tstruct discovery *heads;\n+\n+\tif (for_push)\n+\t\theads = discover_refs(\"git-receive-pack\");\n+\telse\n+\t\theads = discover_refs(\"git-upload-pack\");\n+\n+\tif (heads->proto_git)\n+\t\treturn parse_git_refs(heads);\n+\treturn parse_info_refs(heads);\n+}\n+\n static void output_refs(struct ref *refs)\n {\n \tstruct ref *posn;\n@@ -285,10 +382,10 @@ int main(int argc, const char **argv)\n \t\t\tparse_fetch(&buf, 1);\n \n \t\t} else if (!strcmp(buf.buf, \"list\")) {\n-\t\t\toutput_refs(get_refs());\n+\t\t\toutput_refs(get_refs(0));\n \n \t\t} else if (!strcmp(buf.buf, \"list for-push\")) {\n-\t\t\toutput_refs(get_refs());\n+\t\t\toutput_refs(get_refs(1));\n \n \t\t} else if (!prefixcmp(buf.buf, \"push \")) {\n \t\t\tparse_push(&buf);\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124802","messageId":"1255400715-10508-15-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 14/16] Smart push over HTTP: client side","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:13Z","receivedAt":"2009-10-13T02:25:13Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The git-remote-curl backend detects if the remote server supports\nthe git-receive-pack service, and if so, runs git-send-pack in a\npipe to dump the command and pack data as a single POST request.\n\nThe advertisements from the server that were obtained during the\ndiscovery are passed into git-send-pack before the POST request\nstarts.  This permits git-send-pack to operate largely unmodified.\n\nFor smaller packs (those under 1 MiB) a tranditional POST with a\nContent-Length is used, permitting interaction with any HTTP/1.0\ncompliant server.  The 1 MiB limit is arbitrary, but is sufficent\nto fit most deltas created by human authors against text sources\nwith the occasional small binary file (e.g. few KiB icon image).\n\nFor larger packs which cannot be spooled entirely into the\nhelper's memory space, the POST request requires HTTP/1.1 and\nTransfer-Encoding: chunked.  This permits the client to upload an\nunknown amount of data in one HTTP transaction without needing to\npregenerate the entire pack file locally.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\nCC: Daniel Barkalow <barkalow@iabervon.org>\n---\n Documentation/config.txt |    8 ++\n builtin-send-pack.c      |  116 ++++++++++++++++++++-\n remote-curl.c            |  258 +++++++++++++++++++++++++++++++++++++++++++++-\n send-pack.h              |    3 +-\n sideband.c               |   11 ++-\n transport.c              |    1 +\n 6 files changed, 384 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex cd17814..7130d07 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1089,6 +1089,14 @@ http.maxRequests::\n \tHow many HTTP requests to launch in parallel. Can be overridden\n \tby the 'GIT_HTTP_MAX_REQUESTS' environment variable. Default is 5.\n \n+http.postBuffer::\n+\tMaximum size in bytes of the buffer used by smart HTTP\n+\ttransports when POSTing data to the remote system.\n+\tFor requests larger than this buffer size, HTTP/1.1 and\n+\tTransfer-Encoding: chunked is used to avoid creating a\n+\tmassive pack file locally.  Default is 1 MiB, which is\n+\tsufficient for most requests.\n+\n http.lowSpeedLimit, http.lowSpeedTime::\n \tIf the HTTP transfer speed is less than 'http.lowSpeedLimit'\n \tfor longer than 'http.lowSpeedTime' seconds, the transfer is aborted.\ndiff --git a/builtin-send-pack.c b/builtin-send-pack.c\nindex 37e528e..654c3d4 100644\n--- a/builtin-send-pack.c\n+++ b/builtin-send-pack.c\n@@ -2,9 +2,11 @@\n #include \"commit.h\"\n #include \"refs.h\"\n #include \"pkt-line.h\"\n+#include \"sideband.h\"\n #include \"run-command.h\"\n #include \"remote.h\"\n #include \"send-pack.h\"\n+#include \"quote.h\"\n \n static const char send_pack_usage[] =\n \"git send-pack [--all | --mirror] [--dry-run] [--force] [--receive-pack=<git-receive-pack>] [--verbose] [--thin] [<host>:]<directory> [<ref>...]\\n\"\n@@ -59,7 +61,7 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \tmemset(&po, 0, sizeof(po));\n \tpo.argv = argv;\n \tpo.in = -1;\n-\tpo.out = fd;\n+\tpo.out = args->one_shot_rpc ? -1 : fd;\n \tpo.git_cmd = 1;\n \tif (start_command(&po))\n \t\tdie_errno(\"git pack-objects failed\");\n@@ -83,6 +85,20 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \t}\n \n \tclose(po.in);\n+\n+\tif (args->one_shot_rpc) {\n+\t\tchar *buf = xmalloc(LARGE_PACKET_MAX);\n+\t\twhile (1) {\n+\t\t\tssize_t n = xread(po.out, buf, LARGE_PACKET_MAX);\n+\t\t\tif (n <= 0)\n+\t\t\t\tbreak;\n+\t\t\tsend_sideband(fd, -1, buf, n, LARGE_PACKET_MAX);\n+\t\t}\n+\t\tfree(buf);\n+\t\tclose(po.out);\n+\t\tpo.out = -1;\n+\t}\n+\n \tif (finish_command(&po))\n \t\treturn error(\"pack-objects died with strange error\");\n \treturn 0;\n@@ -303,6 +319,59 @@ static int refs_pushed(struct ref *ref)\n \treturn 0;\n }\n \n+static void print_helper_status(struct ref *ref)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tfor (; ref; ref = ref->next) {\n+\t\tconst char *msg = NULL;\n+\t\tconst char *res;\n+\n+\t\tswitch(ref->status) {\n+\t\tcase REF_STATUS_NONE:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"no match\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_OK:\n+\t\t\tres = \"ok\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_UPTODATE:\n+\t\t\tres = \"ok\";\n+\t\t\tmsg = \"up to date\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"non-fast forward\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_REJECT_NODELETE:\n+\t\tcase REF_STATUS_REMOTE_REJECT:\n+\t\t\tres = \"error\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_EXPECTING_REPORT:\n+\t\tdefault:\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"%s %s\", res, ref->name);\n+\t\tif (ref->remote_status)\n+\t\t\tmsg = ref->remote_status;\n+\t\tif (msg) {\n+\t\t\tstrbuf_addch(&buf, ' ');\n+\t\t\tquote_two_c_style(&buf, \"\", msg, 0);\n+\t\t}\n+\t\tstrbuf_addch(&buf, '\\n');\n+\n+\t\tsafe_write(1, buf.buf, buf.len);\n+\t}\n+\tstrbuf_release(&buf);\n+}\n+\n int send_pack(struct send_pack_args *args,\n \t      int fd[], struct child_process *conn,\n \t      struct ref *remote_refs,\n@@ -310,6 +379,7 @@ int send_pack(struct send_pack_args *args,\n {\n \tint in = fd[0];\n \tint out = fd[1];\n+\tstruct strbuf req_buf = STRBUF_INIT;\n \tstruct ref *ref;\n \tint new_refs;\n \tint ask_for_status_report = 0;\n@@ -391,14 +461,14 @@ int send_pack(struct send_pack_args *args,\n \t\t\tchar *new_hex = sha1_to_hex(ref->new_sha1);\n \n \t\t\tif (ask_for_status_report) {\n-\t\t\t\tpacket_write(out, \"%s %s %s%c%s\",\n+\t\t\t\tpacket_buf_write(&req_buf, \"%s %s %s%c%s\",\n \t\t\t\t\told_hex, new_hex, ref->name, 0,\n \t\t\t\t\t\"report-status\");\n \t\t\t\task_for_status_report = 0;\n \t\t\t\texpect_status_report = 1;\n \t\t\t}\n \t\t\telse\n-\t\t\t\tpacket_write(out, \"%s %s %s\",\n+\t\t\t\tpacket_buf_write(&req_buf, \"%s %s %s\",\n \t\t\t\t\told_hex, new_hex, ref->name);\n \t\t}\n \t\tref->status = expect_status_report ?\n@@ -406,7 +476,17 @@ int send_pack(struct send_pack_args *args,\n \t\t\tREF_STATUS_OK;\n \t}\n \n-\tpacket_flush(out);\n+\tif (args->one_shot_rpc) {\n+\t\tif (!args->dry_run) {\n+\t\t\tpacket_buf_flush(&req_buf);\n+\t\t\tsend_sideband(out, -1, req_buf.buf, req_buf.len, LARGE_PACKET_MAX);\n+\t\t}\n+\t} else {\n+\t\tsafe_write(out, req_buf.buf, req_buf.len);\n+\t\tpacket_flush(out);\n+\t}\n+\tstrbuf_release(&req_buf);\n+\n \tif (new_refs && !args->dry_run) {\n \t\tif (pack_objects(out, remote_refs, extra_have, args) < 0) {\n \t\t\tfor (ref = remote_refs; ref; ref = ref->next)\n@@ -414,11 +494,15 @@ int send_pack(struct send_pack_args *args,\n \t\t\treturn -1;\n \t\t}\n \t}\n+\tif (args->one_shot_rpc && !args->dry_run)\n+\t\tpacket_flush(out);\n \n \tif (expect_status_report)\n \t\tret = receive_status(in, remote_refs);\n \telse\n \t\tret = 0;\n+\tif (args->one_shot_rpc)\n+\t\tpacket_flush(out);\n \n \tif (ret < 0)\n \t\treturn ret;\n@@ -478,6 +562,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tstruct extra_have_objects extra_have;\n \tstruct ref *remote_refs, *local_refs;\n \tint ret;\n+\tint helper_status = 0;\n \tint send_all = 0;\n \tconst char *receivepack = \"git-receive-pack\";\n \tint flags;\n@@ -523,6 +608,14 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\targs.use_thin_pack = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(arg, \"--one-shot-rpc\")) {\n+\t\t\t\targs.one_shot_rpc = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!strcmp(arg, \"--helper-status\")) {\n+\t\t\t\thelper_status = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tusage(send_pack_usage);\n \t\t}\n \t\tif (!dest) {\n@@ -551,7 +644,14 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tconn = git_connect(fd, dest, receivepack, args.verbose ? CONNECT_VERBOSE : 0);\n+\tif (args.one_shot_rpc) {\n+\t\tconn = NULL;\n+\t\tfd[0] = 0;\n+\t\tfd[1] = 1;\n+\t} else {\n+\t\tconn = git_connect(fd, dest, receivepack,\n+\t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n+\t}\n \n \tmemset(&extra_have, 0, sizeof(extra_have));\n \n@@ -575,12 +675,16 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \n \tret = send_pack(&args, fd, conn, remote_refs, &extra_have);\n \n+\tif (helper_status)\n+\t\tprint_helper_status(remote_refs);\n+\n \tclose(fd[1]);\n \tclose(fd[0]);\n \n \tret |= finish_connect(conn);\n \n-\tprint_push_status(dest, remote_refs);\n+\tif (!helper_status)\n+\t\tprint_push_status(dest, remote_refs);\n \n \tif (!args.dry_run && remote) {\n \t\tstruct ref *ref;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 42fd06c..529df42 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -5,17 +5,33 @@\n #include \"walker.h\"\n #include \"http.h\"\n #include \"pkt-line.h\"\n+#include \"sideband.h\"\n #include \"run-command.h\"\n \n \n+static size_t post_buffer_size = 16 * LARGE_PACKET_MAX;\n static struct remote *remote;\n static const char *url;\n static struct walker *walker;\n \n+static int http_options(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(\"http.postbuffer\", var)) {\n+\t\tpost_buffer_size = git_config_int(var, value);\n+\t\tif (post_buffer_size < LARGE_PACKET_MAX)\n+\t\t\tpost_buffer_size = LARGE_PACKET_MAX;\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static void init_walker(void)\n {\n-\tif (!walker)\n+\tif (!walker) {\n \t\twalker = get_http_walker(url, remote);\n+\t\tgit_config(http_options, NULL);\n+\t}\n }\n \n struct discovery {\n@@ -200,6 +216,193 @@ static void output_refs(struct ref *refs)\n \tfree_refs(refs);\n }\n \n+struct rpc_state {\n+\tconst char *service_name;\n+\tchar *service_url;\n+\tchar *hdr_content_type;\n+\tchar *hdr_accept;\n+\tchar *buf;\n+\tsize_t alloc;\n+\tsize_t len;\n+\tsize_t pos;\n+\tint in;\n+\tint out;\n+\tunsigned verbose : 1;\n+};\n+\n+static size_t rpc_out(void *ptr, size_t eltsize,\n+\t\tsize_t nmemb, void *buffer_)\n+{\n+\tsize_t max = eltsize * nmemb;\n+\tstruct rpc_state *state = buffer_;\n+\tsize_t avail = state->len - state->pos;\n+\n+\tif (!avail) {\n+\t\tavail = packet_read_line(state->out, state->buf, state->alloc);\n+\t\tif (!avail)\n+\t\t\treturn 0;\n+\t\tstate->pos = 0;\n+\t\tstate->len = avail;\n+\t}\n+\n+\tif (max < avail);\n+\t\tavail = max;\n+\tmemcpy(ptr, state->buf + state->pos, avail);\n+\tstate->pos += avail;\n+\treturn avail;\n+}\n+\n+static size_t rpc_in(const void *ptr, size_t eltsize,\n+\t\tsize_t nmemb, void *buffer_)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct rpc_state *state = buffer_;\n+\twrite_or_die(state->in, ptr, size);\n+\treturn size;\n+}\n+\n+static int post_rpc(struct rpc_state *state)\n+{\n+\tstruct active_request_slot *slot;\n+\tstruct slot_results results;\n+\tstruct curl_slist *headers = NULL;\n+\tint err = 0, large_request = 0;\n+\n+\t/* Try to load the entire request, if we can fit it into the\n+\t * allocated buffer space we can use HTTP/1.0 and avoid the\n+\t * chunked encoding mess.\n+\t */\n+\twhile (1) {\n+\t\tsize_t left = state->alloc - state->len;\n+\t\tchar *buf = state->buf + state->len;\n+\t\tint n;\n+\n+\t\tif (left < LARGE_PACKET_MAX) {\n+\t\t\tlarge_request = 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tn = packet_read_line(state->out, buf, left);\n+\t\tif (!n)\n+\t\t\tbreak;\n+\t\tstate->len += n;\n+\t}\n+\n+\tslot = get_active_slot();\n+\tslot->results = &results;\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, state->service_url);\n+\n+\theaders = curl_slist_append(headers, state->hdr_content_type);\n+\theaders = curl_slist_append(headers, state->hdr_accept);\n+\n+\tif (large_request) {\n+\t\t/* The request body is large and the size cannot be predicted.\n+\t\t * We must use chunked encoding to send it.\n+\t\t */\n+\t\theaders = curl_slist_append(headers, \"Expect: 100-continue\");\n+\t\theaders = curl_slist_append(headers, \"Transfer-Encoding: chunked\");\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_INFILE, state);\n+\t\tif (state->verbose) {\n+\t\t\tfprintf(stderr, \"POST %s (chunked)\\n\", state->service_name);\n+\t\t\tfflush(stderr);\n+\t\t}\n+\n+\t} else {\n+\t\t/* We know the complete request size in advance, use the\n+\t\t * more normal Content-Length approach.\n+\t\t */\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, state->buf);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, state->len);\n+\t\tif (state->verbose) {\n+\t\t\tfprintf(stderr, \"POST %s (%lu bytes)\\n\",\n+\t\t\t\tstate->service_name, (unsigned long)state->len);\n+\t\t\tfflush(stderr);\n+\t\t}\n+\t}\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, state);\n+\n+\tif (start_active_slot(slot)) {\n+\t\trun_active_slot(slot);\n+\t\tif (results.curl_result != CURLE_OK) {\n+\t\t\terr |= error(\"RPC failed; result=%d, HTTP code = %ld\",\n+\t\t\t\tresults.curl_result, results.http_code);\n+\t\t}\n+\t}\n+\tcurl_slist_free_all(headers);\n+\treturn err;\n+}\n+\n+static int one_shot_rpc_service(const char *service,\n+\tint verbose,\n+\tconst char **client_argv,\n+\tstruct discovery *heads,\n+\tstruct strbuf *result)\n+{\n+\tstruct child_process client;\n+\tstruct rpc_state state;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint err = 0;\n+\n+\tinit_walker();\n+\tmemset(&client, 0, sizeof(client));\n+\tclient.in = -1;\n+\tclient.out = -1;\n+\tclient.git_cmd = 1;\n+\tclient.argv = client_argv;\n+\tif (start_command(&client))\n+\t\tdie(\"%s failed to execute\", client_argv[0]);\n+\tif (heads)\n+\t\twrite_or_die(client.in, heads->buf, heads->len);\n+\n+\tmemset(&state, 0, sizeof(state));\n+\tstate.alloc = post_buffer_size;\n+\tstate.buf = xmalloc(state.alloc);\n+\tstate.in = client.in;\n+\tstate.out = client.out;\n+\tstate.service_name = service;\n+\tstate.verbose = !!verbose;\n+\n+\tstrbuf_addf(&buf, \"%s/%s\", url, service);\n+\tstate.service_url = strbuf_detach(&buf, NULL);\n+\n+\tstrbuf_addf(&buf, \"Content-Type: application/x-%s-request\", service);\n+\tstate.hdr_content_type = strbuf_detach(&buf, NULL);\n+\n+\tstrbuf_addf(&buf, \"Accept: application/x-%s-response\", service);\n+\tstate.hdr_accept = strbuf_detach(&buf, NULL);\n+\n+\twhile (!err) {\n+\t\tint n = packet_read_line(state.out, state.buf, state.alloc);\n+\t\tif (!n)\n+\t\t\tbreak;\n+\t\tstate.pos = 0;\n+\t\tstate.len = n;\n+\t\terr |= post_rpc(&state);\n+\t}\n+\tif (result)\n+\t\tstrbuf_read(result, client.out, 0);\n+\n+\tclose(client.in);\n+\tclose(client.out);\n+\tclient.in = -1;\n+\tclient.out = -1;\n+\n+\terr |= finish_command(&client);\n+\tfree(state.service_url);\n+\tfree(state.hdr_content_type);\n+\tfree(state.hdr_accept);\n+\tfree(state.buf);\n+\tstrbuf_release(&buf);\n+\treturn err;\n+}\n+\n struct fetch_args {\n \tunsigned verbose : 1;\n };\n@@ -289,7 +492,8 @@ static void parse_fetch(struct strbuf *buf, int multiple)\n \n struct push_args {\n \tunsigned dry_run : 1,\n-\t\tverbose : 1;\n+\t\tverbose : 1,\n+\t\tthin : 1;\n };\n \n static int push_dav(struct push_args *args, int nr_spec, char **specs)\n@@ -314,6 +518,51 @@ static int push_dav(struct push_args *args, int nr_spec, char **specs)\n \treturn 0;\n }\n \n+static int push_git(struct discovery *heads,\n+\tstruct push_args *args, int nr_spec, char **specs)\n+{\n+\tstruct strbuf res = STRBUF_INIT;\n+\tconst char **argv;\n+\tint argc = 0, i, err;\n+\n+\targv = xmalloc((10 + nr_spec) * sizeof(char*));\n+\targv[argc++] = \"send-pack\";\n+\targv[argc++] = \"--one-shot-rpc\";\n+\targv[argc++] = \"--helper-status\";\n+\tif (args->thin)\n+\t\targv[argc++] = \"--thin\";\n+\tif (args->dry_run)\n+\t\targv[argc++] = \"--dry-run\";\n+\tif (args->verbose)\n+\t\targv[argc++] = \"--verbose\";\n+\targv[argc++] = url;\n+\tfor (i = 0; i < nr_spec; i++)\n+\t\targv[argc++] = specs[i];\n+\targv[argc++] = NULL;\n+\n+\terr = one_shot_rpc_service(\"git-receive-pack\",\n+\t\targs->verbose,\n+\t\targv, heads, &res);\n+\tif (res.len)\n+\t\tsafe_write(1, res.buf, res.len);\n+\tstrbuf_release(&res);\n+\tfree(argv);\n+\treturn err;\n+}\n+\n+static int push(struct push_args *args, int nr_spec, char **specs)\n+{\n+\tstruct discovery *heads = discover_refs(\"git-receive-pack\");\n+\tint ret;\n+\n+\tif (heads->proto_git)\n+\t\tret = push_git(heads, args, nr_spec, specs);\n+\telse\n+\t\tret = push_dav(args, nr_spec, specs);\n+\tfree_discovery(heads);\n+\treturn ret;\n+}\n+\n static void parse_push(struct strbuf *buf)\n {\n \tchar **specs = NULL;\n@@ -330,6 +579,8 @@ static void parse_push(struct strbuf *buf)\n \t\t\targs.dry_run = 1;\n \t\telse if (!strcmp(buf->buf, \"option verbose\"))\n \t\t\targs.verbose = 1;\n+\t\telse if (!strcmp(buf->buf, \"option thin\"))\n+\t\t\targs.thin = 1;\n \t\telse\n \t\t\tdie(\"http transport does not support %s\", buf->buf);\n \n@@ -340,7 +591,7 @@ static void parse_push(struct strbuf *buf)\n \t\t\tbreak;\n \t} while (1);\n \n-\tif (push_dav(&args, nr_spec, specs))\n+\tif (push(&args, nr_spec, specs))\n \t\texit(128); /* error already reported */\n \tfor (i = 0; i < nr_spec; i++)\n \t\tfree(specs[i]);\n@@ -395,6 +646,7 @@ int main(int argc, const char **argv)\n \t\t\tprintf(\"fetch-multiple\\n\");\n \t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"option dry-run\\n\");\n+\t\t\tprintf(\"option thin\\n\");\n \t\t\tprintf(\"option verbose\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\ndiff --git a/send-pack.h b/send-pack.h\nindex 8b3cf02..a7f4abf 100644\n--- a/send-pack.h\n+++ b/send-pack.h\n@@ -8,7 +8,8 @@ struct send_pack_args {\n \t\tforce_update:1,\n \t\tuse_thin_pack:1,\n \t\tuse_ofs_delta:1,\n-\t\tdry_run:1;\n+\t\tdry_run:1,\n+\t\tone_shot_rpc:1;\n };\n \n int send_pack(struct send_pack_args *args,\ndiff --git a/sideband.c b/sideband.c\nindex 899b1ff..d5ffa1c 100644\n--- a/sideband.c\n+++ b/sideband.c\n@@ -135,9 +135,14 @@ ssize_t send_sideband(int fd, int band, const char *data, ssize_t sz, int packet\n \t\tn = sz;\n \t\tif (packet_max - 5 < n)\n \t\t\tn = packet_max - 5;\n-\t\tsprintf(hdr, \"%04x\", n + 5);\n-\t\thdr[4] = band;\n-\t\tsafe_write(fd, hdr, 5);\n+\t\tif (0 <= band) {\n+\t\t\tsprintf(hdr, \"%04x\", n + 5);\n+\t\t\thdr[4] = band;\n+\t\t\tsafe_write(fd, hdr, 5);\n+\t\t} else {\n+\t\t\tsprintf(hdr, \"%04x\", n + 4);\n+\t\t\tsafe_write(fd, hdr, 4);\n+\t\t}\n \t\tsafe_write(fd, p, n);\n \t\tp += n;\n \t\tsz -= n;\ndiff --git a/transport.c b/transport.c\nindex 6d9652d..2ff1650 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -731,6 +731,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\t\t\t NULL);\n \t}\n \n+\tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n \targs.force_update = !!(flags & TRANSPORT_PUSH_FORCE);\n \targs.use_thin_pack = data->thin;\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124803","messageId":"1255400715-10508-16-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 15/16] Smart fetch over HTTP: client side","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:14Z","receivedAt":"2009-10-13T02:25:14Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The git-remote-curl backend detects if the remote server supports\nthe git-upload-pack service, and if so, runs git-fetch-pack locally\nin a pipe to generate the want/have commands.\n\nThe advertisements from the server that were obtained during the\ndiscovery are passed into git-fetch-pack before the POST request\nstarts, permitting server capability discovery and enablement.\n\nCommon objects that are discovered are appended onto the request as\nhave lines and are sent again on the next request.  This allows the\nremote side to reinitialize its in-memory list of common objects\nduring the next request.\n\nBecause all requests are relatively short, below git-remote-curl's\n1 MiB buffer limit, requests will use the standard Content-Length\nheader and be valid HTTP/1.0 POST requests.  This makes the fetch\nclient more tolerant of proxy servers which don't support HTTP/1.1\nor the chunked transfer encoding.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\nCC: Daniel Barkalow <barkalow@iabervon.org>\n---\n builtin-fetch-pack.c |  110 ++++++++++++++++++++++++++++++++++++++++++--------\n fetch-pack.h         |    3 +-\n remote-curl.c        |   75 +++++++++++++++++++++++++++++++++-\n 3 files changed, 168 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex b68b3eb..fba0c74 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -165,6 +165,24 @@ enum ack_type {\n \tACK_ready\n };\n \n+static void consume_shallow_list(int fd)\n+{\n+\tif (args.one_shot_rpc && args.depth > 0) {\n+\t\t/* If we sent a depth we will get back \"duplicate\"\n+\t\t * shallow and unshallow commands every time there\n+\t\t * is a block of have lines exchanged.\n+\t\t */\n+\t\tchar line[1000];\n+\t\twhile (packet_read_line(fd, line, sizeof(line))) {\n+\t\t\tif (!prefixcmp(line, \"shallow \"))\n+\t\t\t\tcontinue;\n+\t\t \tif (!prefixcmp(line, \"unshallow \"))\n+\t\t\t\tcontinue;\n+\t\t\tdie(\"git fetch-pack: expected shallow list\");\n+\t\t}\n+\t}\n+}\n+\n static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n {\n \tstatic char line[1000];\n@@ -190,6 +208,15 @@ static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n \tdie(\"git fetch_pack: expected ACK/NAK, got '%s'\", line);\n }\n \n+static void send_request(int fd, struct strbuf *buf)\n+{\n+\tif (args.one_shot_rpc) {\n+\t\tsend_sideband(fd, -1, buf->buf, buf->len, LARGE_PACKET_MAX);\n+\t\tpacket_flush(fd);\n+\t} else\n+\t\tsafe_write(fd, buf->buf, buf->len);\n+}\n+\n static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t       struct ref *refs)\n {\n@@ -199,7 +226,10 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tunsigned in_vain = 0;\n \tint got_continue = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n+\tsize_t state_len = 0;\n \n+\tif (args.one_shot_rpc && multi_ack == 1)\n+\t\tdie(\"--one-shot-rpc requires multi_ack_2\");\n \tif (marked)\n \t\tfor_each_ref(clear_marks, NULL);\n \tmarked = 1;\n@@ -256,13 +286,13 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tif (args.depth > 0)\n \t\tpacket_buf_write(&req_buf, \"deepen %d\", args.depth);\n \tpacket_buf_flush(&req_buf);\n-\n-\tsafe_write(fd[1], req_buf.buf, req_buf.len);\n+\tstate_len = req_buf.len;\n \n \tif (args.depth > 0) {\n \t\tchar line[1024];\n \t\tunsigned char sha1[20];\n \n+\t\tsend_request(fd[1], &req_buf);\n \t\twhile (packet_read_line(fd[0], line, sizeof(line))) {\n \t\t\tif (!prefixcmp(line, \"shallow \")) {\n \t\t\t\tif (get_sha1_hex(line + 8, sha1))\n@@ -284,28 +314,40 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t\t}\n \t\t\tdie(\"expected shallow/unshallow, got %s\", line);\n \t\t}\n+\t} else if (!args.one_shot_rpc)\n+\t\tsend_request(fd[1], &req_buf);\n+\n+\tif (!args.one_shot_rpc) {\n+\t\t/* If we aren't using the stateless one-shot-rpc\n+\t\t * interface we don't need to retain the headers.\n+\t\t */\n+\t\tstrbuf_setlen(&req_buf, 0);\n+\t\tstate_len = 0;\n \t}\n \n \tflushes = 0;\n \tretval = -1;\n \twhile ((sha1 = get_rev())) {\n-\t\tpacket_write(fd[1], \"have %s\\n\", sha1_to_hex(sha1));\n+\t\tpacket_buf_write(&req_buf, \"have %s\\n\", sha1_to_hex(sha1));\n \t\tif (args.verbose)\n \t\t\tfprintf(stderr, \"have %s\\n\", sha1_to_hex(sha1));\n \t\tin_vain++;\n \t\tif (!(31 & ++count)) {\n \t\t\tint ack;\n \n-\t\t\tpacket_flush(fd[1]);\n+\t\t\tpacket_buf_flush(&req_buf);\n+\t\t\tsend_request(fd[1], &req_buf);\n+\t\t\tstrbuf_setlen(&req_buf, state_len);\n \t\t\tflushes++;\n \n \t\t\t/*\n \t\t\t * We keep one window \"ahead\" of the other side, and\n \t\t\t * will wait for an ACK only on the next one\n \t\t\t */\n-\t\t\tif (count == 32)\n+\t\t\tif (!args.one_shot_rpc && count == 32)\n \t\t\t\tcontinue;\n \n+\t\t\tconsume_shallow_list(fd[0]);\n \t\t\tdo {\n \t\t\t\tack = get_ack(fd[0], result_sha1);\n \t\t\t\tif (args.verbose && ack)\n@@ -322,6 +364,17 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t\t\tcase ACK_continue: {\n \t\t\t\t\tstruct commit *commit =\n \t\t\t\t\t\tlookup_commit(result_sha1);\n+\t\t\t\t\tif (args.one_shot_rpc\n+\t\t\t\t\t && ack == ACK_common\n+\t\t\t\t\t && !(commit->object.flags & COMMON)) {\n+\t\t\t\t\t\t/* We need to replay the have for this object\n+\t\t\t\t\t\t * on the next RPC request so the peer knows\n+\t\t\t\t\t\t * it is in common with us.\n+\t\t\t\t\t\t */\n+\t\t\t\t\t\tconst char *hex = sha1_to_hex(result_sha1);\n+\t\t\t\t\t\tpacket_buf_write(&req_buf, \"have %s\\n\", hex);\n+\t\t\t\t\t\tstate_len = req_buf.len;\n+\t\t\t\t\t}\n \t\t\t\t\tmark_common(commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tin_vain = 0;\n@@ -339,7 +392,8 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t}\n \t}\n done:\n-\tpacket_write(fd[1], \"done\\n\");\n+\tpacket_buf_write(&req_buf, \"done\\n\");\n+\tsend_request(fd[1], &req_buf);\n \tif (args.verbose)\n \t\tfprintf(stderr, \"done\\n\");\n \tif (retval != 0) {\n@@ -348,6 +402,7 @@ done:\n \t}\n \tstrbuf_release(&req_buf);\n \n+\tconsume_shallow_list(fd[0]);\n \twhile (flushes || multi_ack) {\n \t\tint ack = get_ack(fd[0], result_sha1);\n \t\tif (ack) {\n@@ -672,6 +727,8 @@ static struct ref *do_fetch_pack(int fd[2],\n \t\t\t */\n \t\t\twarning(\"no common commits\");\n \n+\tif (args.one_shot_rpc)\n+\t\tpacket_flush(fd[1]);\n \tif (get_pack(fd, pack_lockfile))\n \t\tdie(\"git fetch-pack: fetch failed.\");\n \n@@ -742,6 +799,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tstruct ref *ref = NULL;\n \tchar *dest = NULL, **heads;\n \tint fd[2];\n+\tchar *pack_lockfile = NULL;\n+\tchar **pack_lockfile_ptr = NULL;\n \tstruct child_process *conn;\n \n \tnr_heads = 0;\n@@ -791,6 +850,15 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\targs.no_progress = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(\"--one-shot-rpc\", arg)) {\n+\t\t\t\targs.one_shot_rpc = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!strcmp(\"--lock-pack\", arg)) {\n+\t\t\t\targs.lock_pack = 1;\n+\t\t\t\tpack_lockfile_ptr = &pack_lockfile;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tusage(fetch_pack_usage);\n \t\t}\n \t\tdest = (char *)arg;\n@@ -801,19 +869,27 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tif (!dest)\n \t\tusage(fetch_pack_usage);\n \n-\tconn = git_connect(fd, (char *)dest, args.uploadpack,\n-\t\t\t   args.verbose ? CONNECT_VERBOSE : 0);\n-\tif (conn) {\n-\t\tget_remote_heads(fd[0], &ref, 0, NULL, 0, NULL);\n-\n-\t\tref = fetch_pack(&args, fd, conn, ref, dest, nr_heads, heads, NULL);\n-\t\tclose(fd[0]);\n-\t\tclose(fd[1]);\n-\t\tif (finish_connect(conn))\n-\t\t\tref = NULL;\n+\tif (args.one_shot_rpc) {\n+\t\tconn = NULL;\n+\t\tfd[0] = 0;\n+\t\tfd[1] = 1;\n \t} else {\n-\t\tref = NULL;\n+\t\tconn = git_connect(fd, (char *)dest, args.uploadpack,\n+\t\t\t\t   args.verbose ? CONNECT_VERBOSE : 0);\n+\t}\n+\n+\tget_remote_heads(fd[0], &ref, 0, NULL, 0, NULL);\n+\n+\tref = fetch_pack(&args, fd, conn, ref, dest,\n+\t\tnr_heads, heads, pack_lockfile_ptr);\n+\tif (pack_lockfile) {\n+\t\tprintf(\"lock %s\\n\", pack_lockfile);\n+\t\tfflush(stdout);\n \t}\n+\tclose(fd[0]);\n+\tclose(fd[1]);\n+\tif (finish_connect(conn))\n+\t\tref = NULL;\n \tret = !ref;\n \n \tif (!ret && nr_heads) {\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 8bd9c32..ee6b17f 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -13,7 +13,8 @@ struct fetch_pack_args\n \t\tfetch_all:1,\n \t\tverbose:1,\n \t\tno_progress:1,\n-\t\tinclude_tag:1;\n+\t\tinclude_tag:1,\n+\t\tone_shot_rpc:1;\n };\n \n struct ref *fetch_pack(struct fetch_pack_args *args,\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 529df42..31d1d34 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -404,7 +404,10 @@ static int one_shot_rpc_service(const char *service,\n }\n \n struct fetch_args {\n-\tunsigned verbose : 1;\n+\tint depth;\n+\tunsigned verbose : 1,\n+\t\tthin : 1,\n+\t\tfollowtags : 1;\n };\n \n static int fetch_dumb(struct fetch_args *args,\n@@ -413,6 +416,8 @@ static int fetch_dumb(struct fetch_args *args,\n \tchar **targets = xmalloc(nr_heads * sizeof(char*));\n \tint ret, i;\n \n+\tif (args->depth)\n+\t\tdie(\"dumb http transport does not support --depth\");\n \tfor (i = 0; i < nr_heads; i++)\n \t\ttargets[i] = xstrdup(sha1_to_hex(to_fetch[i]->old_sha1));\n \n@@ -431,6 +436,64 @@ static int fetch_dumb(struct fetch_args *args,\n \treturn ret ? error(\"Fetch failed.\") : 0;\n }\n \n+static int fetch_git(struct fetch_args *args,\n+\tstruct discovery *heads,\n+\tint nr_heads, struct ref **to_fetch)\n+{\n+\tstruct strbuf res = STRBUF_INIT;\n+\tchar *depth_arg = NULL;\n+\tconst char **argv;\n+\tint argc = 0, i, err;\n+\n+\targv = xmalloc((15 + nr_heads) * sizeof(char*));\n+\targv[argc++] = \"fetch-pack\";\n+\targv[argc++] = \"--one-shot-rpc\";\n+\targv[argc++] = \"--lock-pack\";\n+\tif (args->followtags)\n+\t\targv[argc++] = \"--include-tag\";\n+\tif (args->thin)\n+\t\targv[argc++] = \"--thin\";\n+\tif (args->verbose) {\n+\t\targv[argc++] = \"-v\";\n+\t\targv[argc++] = \"-v\";\n+\t}\n+\tif (args->depth) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"--depth=%d\", args->depth);\n+\t\tdepth_arg = strbuf_detach(&buf, NULL);\n+\t\targv[argc++] = depth_arg;\n+\t}\n+\targv[argc++] = url;\n+\tfor (i = 0; i < nr_heads; i++) {\n+\t\tstruct ref *ref = to_fetch[i];\n+\t\tif (!ref->name || !*ref->name)\n+\t\t\tdie(\"cannot fetch by sha1 over smart http\");\n+\t\targv[argc++] = ref->name;\n+\t}\n+\targv[argc++] = NULL;\n+\n+\terr = one_shot_rpc_service(\"git-upload-pack\",\n+\t\targs->verbose,\n+\t\targv, heads, &res);\n+\tif (res.len)\n+\t\tsafe_write(1, res.buf, res.len);\n+\tstrbuf_release(&res);\n+\n+\tfree(argv);\n+\tfree(depth_arg);\n+\treturn err;\n+}\n+\n+static int fetch(struct fetch_args *args,\n+\tint nr_heads, struct ref **to_fetch)\n+{\n+\tstruct discovery *d = discover_refs(\"git-upload-pack\");\n+\tif (d->proto_git)\n+\t\treturn fetch_git(args, d, nr_heads, to_fetch);\n+\telse \n+\t\treturn fetch_dumb(args, nr_heads, to_fetch);\n+}\n+\n static void parse_fetch(struct strbuf *buf, int multiple)\n {\n \tstruct ref **to_fetch = NULL;\n@@ -470,6 +533,12 @@ static void parse_fetch(struct strbuf *buf, int multiple)\n \t\t}\n \t\telse if (!strcmp(buf->buf, \"option verbose\"))\n \t\t\targs.verbose = 1;\n+\t\telse if (!strcmp(buf->buf, \"option followtags\"))\n+\t\t\targs.followtags = 1;\n+\t\telse if (!strcmp(buf->buf, \"option thin\"))\n+\t\t\targs.thin = 1;\n+\t\telse if (!prefixcmp(buf->buf, \"option depth \"))\n+\t\t\targs.depth = atoi(buf->buf + strlen(\"option depth \"));\n \t\telse\n \t\t\tdie(\"http transport does not support %s\", buf->buf);\n \n@@ -480,7 +549,7 @@ static void parse_fetch(struct strbuf *buf, int multiple)\n \t\t\tbreak;\n \t} while (1);\n \n-\tif (fetch_dumb(&args, nr_heads, to_fetch))\n+\tif (fetch(&args, nr_heads, to_fetch))\n \t\texit(128); /* error already reported */\n \tfree_refs(list_head);\n \tfree(to_fetch);\n@@ -645,7 +714,9 @@ int main(int argc, const char **argv)\n \t\t\tprintf(\"fetch\\n\");\n \t\t\tprintf(\"fetch-multiple\\n\");\n \t\t\tprintf(\"push\\n\");\n+\t\t\tprintf(\"option depth\\n\");\n \t\t\tprintf(\"option dry-run\\n\");\n+\t\t\tprintf(\"option followtags\\n\");\n \t\t\tprintf(\"option thin\\n\");\n \t\t\tprintf(\"option verbose\\n\");\n \t\t\tprintf(\"\\n\");\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124799","messageId":"1255400715-10508-17-git-send-email-spearce@spearce.org","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v2 16/16] Smart HTTP fetch: gzip requests","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T02:25:15Z","receivedAt":"2009-10-13T02:25:15Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The upload-pack requests are mostly plain text and they compress\nrather well.  Deflating them with Content-Encoding: gzip can easily\ndrop the size of the request by 50%, reducing the amount of data\nto transfer as we negotiate the common commits.\n\nSigned-off-by: Shawn O. Pearce <spearce@spearce.org>\nCC: Daniel Barkalow <barkalow@iabervon.org>\n---\n remote-curl.c |   52 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 50 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 31d1d34..d53215d 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -261,11 +261,12 @@ static size_t rpc_in(const void *ptr, size_t eltsize,\n \treturn size;\n }\n \n-static int post_rpc(struct rpc_state *state)\n+static int post_rpc(struct rpc_state *state, int use_gzip)\n {\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n \tstruct curl_slist *headers = NULL;\n+\tunsigned char *gzip_body = NULL;\n \tint err = 0, large_request = 0;\n \n \t/* Try to load the entire request, if we can fit it into the\n@@ -279,6 +280,7 @@ static int post_rpc(struct rpc_state *state)\n \n \t\tif (left < LARGE_PACKET_MAX) {\n \t\t\tlarge_request = 1;\n+\t\t\tuse_gzip = 0;\n \t\t\tbreak;\n \t\t}\n \n@@ -311,6 +313,48 @@ static int post_rpc(struct rpc_state *state)\n \t\t\tfflush(stderr);\n \t\t}\n \n+\t} else if (use_gzip && 1024 < state->len) {\n+\t\t/* The client backend isn't giving us compressed data so\n+\t\t * we can try to deflate it ourselves, this may save on.\n+\t\t * the transfer time.\n+\t\t */\n+\t\tsize_t size;\n+\t\tz_stream stream;\n+\t\tint ret;\n+\n+\t\tmemset(&stream, 0, sizeof(stream));\n+\t\tret = deflateInit2(&stream, Z_DEFAULT_COMPRESSION,\n+\t\t\t\tZ_DEFLATED, (15 + 16),\n+\t\t\t\t8, Z_DEFAULT_STRATEGY);\n+\t\tif (ret != Z_OK)\n+\t\t\tdie(\"cannot deflate request; zlib init error %d\", ret);\n+\t\tsize = deflateBound(&stream, state->len);\n+\t\tgzip_body = xmalloc(size);\n+\n+\t\tstream.next_in = (unsigned char *)state->buf;\n+\t\tstream.avail_in = state->len;\n+\t\tstream.next_out = gzip_body;\n+\t\tstream.avail_out = size;\n+\n+\t\tret = deflate(&stream, Z_FINISH);\n+\t\tif (ret != Z_STREAM_END)\n+\t\t\tdie(\"cannot deflate request; zlib deflate error %d\", ret);\n+\n+\t\tret = deflateEnd(&stream);\n+\t\tif (ret != Z_OK)\n+\t\t\tdie(\"cannot deflate request; zlib end error %d\", ret);\n+\n+\t\tsize = stream.total_out;\n+\n+\t\theaders = curl_slist_append(headers, \"Content-Encoding: gzip\");\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, size);\n+\n+\t\tif (state->verbose) {\n+\t\t\tfprintf(stderr, \"POST %s (gzip %lu to %lu bytes)\\n\",\n+\t\t\t\tstate->service_name, state->len, size);\n+\t\t\tfflush(stderr);\n+\t\t}\n \t} else {\n \t\t/* We know the complete request size in advance, use the\n \t\t * more normal Content-Length approach.\n@@ -336,11 +380,13 @@ static int post_rpc(struct rpc_state *state)\n \t\t}\n \t}\n \tcurl_slist_free_all(headers);\n+\tfree(gzip_body);\n \treturn err;\n }\n \n static int one_shot_rpc_service(const char *service,\n \tint verbose,\n+\tint use_gzip,\n \tconst char **client_argv,\n \tstruct discovery *heads,\n \tstruct strbuf *result)\n@@ -384,7 +430,7 @@ static int one_shot_rpc_service(const char *service,\n \t\t\tbreak;\n \t\tstate.pos = 0;\n \t\tstate.len = n;\n-\t\terr |= post_rpc(&state);\n+\t\terr |= post_rpc(&state, use_gzip);\n \t}\n \tif (result)\n \t\tstrbuf_read(result, client.out, 0);\n@@ -474,6 +520,7 @@ static int fetch_git(struct fetch_args *args,\n \n \terr = one_shot_rpc_service(\"git-upload-pack\",\n \t\targs->verbose,\n+\t\t1 /* gzip request */,\n \t\targv, heads, &res);\n \tif (res.len)\n \t\tsafe_write(1, res.buf, res.len);\n@@ -611,6 +658,7 @@ static int push_git(struct discovery *heads,\n \n \terr = one_shot_rpc_service(\"git-receive-pack\",\n \t\targs->verbose,\n+\t\t0 /* no gzip */,\n \t\targv, heads, &res);\n \tif (res.len)\n \t\tsafe_write(1, res.buf, res.len);\n-- \n1.6.5.52.g0ff2e\n"},{"id":"124805","messageId":"4AD3F7C5.2060203@gmail.com","threadId":"21220","inReplyTo":"1255400715-10508-1-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 00/16] Return of smart HTTP","fromName":"eduard stefan","fromEmail":"eduard.stefan@gmail.com","sentAt":"2009-10-13T03:45:09Z","receivedAt":"2009-10-13T03:45:09Z","isPatch":true,"sender":{"key":"eduard.stefan@gmail.com","avatar":null},"body":"Shawn O. Pearce wrote:\n> The series has gotten a lot larger since my last posting, but I have\n> what appears to be a fully working client *AND* server implementation\n> for both fetch and push, and the client should be supporting deeping\n> shallow repositories over the smart variant of HTTP.\n> \n> I've dropped the documentation patch from the series for now as\n> I have quite a few edits queued up from folks in the last round\n> (thanks for those!) that I have not yet applied.  So there is no\n> point in sending that particular patch again.\n\nI have apllied you patches on top of Git 1.6.5 release,\nand they solved my http cloning crashes on windows\n(msysGit environment).\n\nNow git http cloning works just fine for local http servers,\nbut on GitHub I get an 502 error:\n\"error: Unable to get pack file\nhttp://github.com/loudej/spark.git/objects/pack/\npack-1bc121e71e2847622f814603ddb34444bfc6a16c.pack\nThe requested URL returned error: 502\"\nwhich seems to be more like a GitHub problem,\nsince that URL returns an error when accessing it with a browser.\n\nHave a nice day,\n  Eduard\n"},{"id":"124806","messageId":"alpine.LNX.2.00.0910122326130.32515@iabervon.org","threadId":"21220","inReplyTo":"1255400715-10508-8-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 07/16] remote-helpers: Fetch more than one ref in a batch","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-10-13T03:56:43Z","receivedAt":"2009-10-13T03:56:43Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 12 Oct 2009, Shawn O. Pearce wrote:\n\n> Some network protocols (e.g. native git://) are able to fetch more\n> than one ref at a time and reduce the overall transfer cost by\n> combining the requests into a single exchange.  Instead of feeding\n> each fetch request one at a time to the helper, feed all of them\n> at once so the helper can decide whether or not it should batch them.\n> \n> Because 'fetch' was already released in 1.6.5 we introduce the new\n> fetch-multiple capability/command to signal that the helper wants\n> to use batch oriented approach to fetching refs.\n\nIn 1.6.5, there's no way to call a helper other than git-remote-curl, and \nno way to call git-remote-curl unless 1.6.5 was built with it. So I think \nthe protocol is not set in stone quite yet. It's documentated for being an \nAPI, and it's supposed to be that, but it's not quite there in this \nversion.\n\nI think it should be generally a good idea to have a start signal and an \nend signal for a block of fetches (and, with the foreign stuff, it would \nbe useful to have transport-helper tell the helper process when it was \ndone making requests, so the helper process could tell the gfi process to \nexit and stop consuming the helper process's output). At worst, if the \nhelper doesn't care, it can just ignore this information.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"124808","messageId":"alpine.LNX.2.00.0910122357230.32515@iabervon.org","threadId":"21220","inReplyTo":"1255400715-10508-9-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 08/16] remote-helpers: Support custom transport options","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-10-13T04:23:23Z","receivedAt":"2009-10-13T04:23:23Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 12 Oct 2009, Shawn O. Pearce wrote:\n\n> Some transports, like the native pack transport implemented by\n> fetch-pack, support useful features like depth or include tags.\n> These should be exposed if the underlying helper knows how to\n> use them and is based upon the same infrastructure.\n>\n> Helpers must advertise the options they support, any attempt\n> to set an unsupported option will cause a failure.\n> \n> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>\n> CC: Daniel Barkalow <barkalow@iabervon.org>\n> ---\n>  Documentation/git-remote-helpers.txt |   20 ++++++++++\n>  remote-curl.c                        |   16 ++++++-\n>  transport-helper.c                   |   70 ++++++++++++++++++++++++++++++++++\n>  3 files changed, 103 insertions(+), 3 deletions(-)\n> \n> diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\n> index e10ce99..334ab30 100644\n> --- a/Documentation/git-remote-helpers.txt\n> +++ b/Documentation/git-remote-helpers.txt\n> @@ -46,6 +46,7 @@ Supported if the helper has the \"fetch\" capability.\n>  'fetch-multiple'::\n>  \tFetches multiple objects at once.  The fetch-multiple\n>  \tcommand is followed by one or more 'fetch' lines as above,\n> +\tzero or more 'option' lines for the supported options,\n>  \tand then a blank line to terminate the batch.  Outputs a\n>  \tsingle blank line when the entire batch is complete.\n>  \tOptionally may output a 'lock <file>' line indicating a\n> @@ -69,6 +70,9 @@ CAPABILITIES\n>  'fetch-multiple'::\n>  \tThis helper supports the 'fetch-multiple' command.\n>  \n> +'option' <name>::\n> +\tThis helper supports the option <name> under fetch-multiple.\n> +\n\nI'm a bit surprised that the options only apply in a fetch-multiple \nsection, rather than getting set at the beginning and applying to \neverything for that run. At least, I think an \"option\" command should be \nuseable outside of a fetch-multiple (or possible future grouping \nconstruct) and have global scope.\n\n>  REF LIST ATTRIBUTES\n>  -------------------\n>  \n> @@ -76,10 +80,26 @@ None are defined yet, but the caller must accept any which are supplied.\n>  \n>  FETCH OPTIONS\n>  -------------\n> +To enable an option the helper must list it in 'capabilities'.\n>  \n>  'option verbose'::\n>  \tPrint more verbose activity messages to stderr.\n\nI think you mis-split the above part; your previoud patch declared this \noption without declaring any way to use it. Might be worth allowing \nmultiple \"verboses\" and \"quiet\" or \"option verbosity quiet\"/\"option \nverbosity verbose verbose\".\n\n> +'option uploadpack' <command>::\n> +\tThe program to use on the remote side to generate a pack.\n\nI sort of feel like the helper ought to read this one out of the config \nfile itself if it wants it. In general, it would be good to have \ntransport.c and remote.c out of the business of knowing this sort of \nprotocol-specific (albiet specific now to two protocols) information. (Of \ncourse, the native protocol's transport methods are in transport.c, so \nthat's there, but I'd like to move that to a transport-native.c someday.)\n\n> +'option depth' <depth>::\n> +\tDeepen the history of a shallow repository.\n> +\n> +'option keep'::\n> +\tKeep the transferred pack(s) with .keep files.\n> +\n> +'option followtags'::\n> +\tAggressively fetch annotated tags if possible.\n\nI assume this means to fetch tags which annotate objects we have or are \nfetching? (As opposed to fetching any annotated tag we could possibly \nfetch, even if we don't otherwise care about the tag or the thing it \ntags.) It's obvious in the context of git's config options, but I'd like \nthis document to avoid assuming that context, and the option could apply \nmore generally.\n\n> +\n> +'option thin'::\n> +\tTransfer the data as a thin pack if possible.\n\nDoes anyone still use non-default thinness? \n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"124809","messageId":"20091013044141.GA18961@glandium.org","threadId":"21220","inReplyTo":"1255400715-10508-10-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 09/16] Move WebDAV HTTP push under remote-curl","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2009-10-13T04:41:41Z","receivedAt":"2009-10-13T04:41:41Z","isPatch":true,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Mon, Oct 12, 2009 at 07:25:08PM -0700, Shawn O. Pearce wrote:\n> The remote helper interface now supports the push capability,\n> which can be used to ask the implementation to push one or more\n> specs to the remote repository.  For remote-curl we implement this\n> by calling the existing WebDAV based git-http-push executable.\n> \n> Internally the helper interface uses the push_refs transport hook\n> so that the complexity of the refspec parsing and matching can be\n> reused between remote implementations.  When possible however the\n> helper protocol uses source ref name rather than the source SHA-1,\n> thereby allowing the helper to access this name if it is useful.\n\nIt's been a while I haven't followed changes in the remote code but this\nlooks nice, though I haven't checked thoroughly. I guess the next step\nwould be to kill http-push as an external program.\n\nMike\n"},{"id":"124815","messageId":"7vbpkb3iby.fsf@alter.siamese.dyndns.org","threadId":"21220","inReplyTo":"4AD3F7C5.2060203@gmail.com","subject":"Re: [RFC PATCH v2 00/16] Return of smart HTTP","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-13T06:42:57Z","receivedAt":"2009-10-13T06:42:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"eduard stefan <eduard.stefan@gmail.com> writes:\n\n> I have apllied you patches on top of Git 1.6.5 release,\n> and they solved my http cloning crashes on windows\n> (msysGit environment).\n\nUnfortunately that is not a very good news.\n\nAs there is no chance Shawn's new series will be applied to 'maint' for\nupdating 1.6.5.X series, somebody needs to address the breakage without\nintroducing the whole new HTTP code.  Is the breakage you are seeing\nlimited only to msysgit environment, or does the same breakage appear on\nother platforms?\n"},{"id":"124823","messageId":"4AD42C52.80205@viscovery.net","threadId":"21220","inReplyTo":"1255400715-10508-2-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 01/16] pkt-line: Add strbuf based functions","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-10-13T07:29:22Z","receivedAt":"2009-10-13T07:29:22Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Shawn O. Pearce schrieb:\n> -int packet_read_line(int fd, char *buffer, unsigned size)\n> +static int packet_length(unsigned *ret_len, const char *linelen)\n>  {\n>  \tint n;\n> -\tunsigned len;\n> -\tchar linelen[4];\n> -\n> -\tsafe_read(fd, linelen, 4);\n> +\tunsigned len = 0;\n>  \n> -\tlen = 0;\n>  \tfor (n = 0; n < 4; n++) {\n>  \t\tunsigned char c = linelen[n];\n>  \t\tlen <<= 4;\n> @@ -96,8 +116,20 @@ int packet_read_line(int fd, char *buffer, unsigned size)\n>  \t\t\tlen += c - 'A' + 10;\n>  \t\t\tcontinue;\n>  \t\t}\n> -\t\tdie(\"protocol error: bad line length character\");\n> +\t\treturn -1;\n>  \t}\n> +\t*ret_len = len;\n> +\treturn 0;\n> +}\n\nlen can be signed: Valid lengths fit into a signed int. Then you can\n'return len;' on success and 'return -1;' on failure and don't need return\nthe result by reference. packet_read_line() ultimately converts it to int\nanyway:\n\n> +int packet_read_line(int fd, char *buffer, unsigned size)\n> +{\n> +\tunsigned len;\n> +\tchar linelen[4];\n> +\n> +\tsafe_read(fd, linelen, 4);\n> +\tif (packet_length(&len, linelen))\n> +\t\tdie(\"protocol error: bad line length character\");\n>  \tif (!len)\n>  \t\treturn 0;\n>  \tlen -= 4;\n> @@ -107,3 +139,28 @@ int packet_read_line(int fd, char *buffer, unsigned size)\n>  \tbuffer[len] = 0;\n>  \treturn len;\n>  }\n\n-- Hannes\n"},{"id":"124822","messageId":"4AD42C87.6000205@viscovery.net","threadId":"21220","inReplyTo":"1255400715-10508-13-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 12/16] Smart fetch and push over HTTP: server side","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-10-13T07:30:15Z","receivedAt":"2009-10-13T07:30:15Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Shawn O. Pearce schrieb:\n> diff --git a/http-backend.c b/http-backend.c\n> index 39cfd25..adb3256 100644\n> --- a/http-backend.c\n> +++ b/http-backend.c\n\n#include \"run-command.h\" is missing here because you added it already in\npatch 10/16 unnecessarily.\n\n> +\tif (start_command(&cld))\n> +\t\tdie_errno(\"Cannot start %s\", argv[0]);\n> ...\n> +\tif (finish_command(&cld))\n> +\t\tdie(\"%s terminated with error\", argv[0]);\n\nstart_command and finish_command already write an error message for you\nthat includes argv[0] and errno. You can just exit(1) here.\n\n-- Hannes\n"},{"id":"124829","messageId":"7vocobzo19.fsf@alter.siamese.dyndns.org","threadId":"21220","inReplyTo":"1255400715-10508-17-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 16/16] Smart HTTP fetch: gzip requests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-13T08:38:42Z","receivedAt":"2009-10-13T08:38:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" <spearce@spearce.org> writes:\n\n> diff --git a/remote-curl.c b/remote-curl.c\n> index 31d1d34..d53215d 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -261,11 +261,12 @@ static size_t rpc_in(const void *ptr, size_t eltsize,\n>  \treturn size;\n>  }\n>  \n> -static int post_rpc(struct rpc_state *state)\n> +static int post_rpc(struct rpc_state *state, int use_gzip)\n>  {\n>  \tstruct active_request_slot *slot;\n>  \tstruct slot_results results;\n>  \tstruct curl_slist *headers = NULL;\n> +\tunsigned char *gzip_body = NULL;\n>  \tint err = 0, large_request = 0;\n>  \n>  \t/* Try to load the entire request, if we can fit it into the\n> @@ -311,6 +313,48 @@ static int post_rpc(struct rpc_state *state)\n> ...\n> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);\n\ncc1: warnings being treated as errors\nIn function 'post_rpc',\n    inlined from 'one_shot_rpc_service' at remote-curl.c:433:\nremote-curl.c:350: error: call to '_curl_easy_setopt_err_postfields' declared with attribute warning: curl_easy_setopt expects a void* or char* argument for this option\nmake: *** [remote-curl.o] Error 1\n"},{"id":"124840","messageId":"4AD45CCD.1000509@viscovery.net","threadId":"21220","inReplyTo":"1255400715-10508-11-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 10/16] Git-aware CGI to provide dumb HTTP transport","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-10-13T10:56:13Z","receivedAt":"2009-10-13T10:56:13Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Shawn O. Pearce schrieb:\n> +int main(int argc, char **argv)\n> +{\n> +\tchar *dir = getenv(\"PATH_TRANSLATED\");\n> +\tchar *input_method = getenv(\"REQUEST_METHOD\");\n> +\tstruct service_cmd *cmd = NULL;\n> +\tchar *cmd_arg = NULL;\n> +\tint i;\n> +\n\nPlease insert here:\n\n\tgit_extract_argv0_path(argv[0]);\n\n> +\tset_die_routine(die_webcgi);\n> ...\n\n-- Hannes\n"},{"id":"124841","messageId":"94a0d4530910130402j597f156dr9ed7eb38a0077252@mail.gmail.com","threadId":"21220","inReplyTo":"1255400715-10508-15-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 14/16] Smart push over HTTP: client side","fromName":"Felipe Contreras","fromEmail":"felipe.contreras@gmail.com","sentAt":"2009-10-13T11:02:46Z","receivedAt":"2009-10-13T11:02:46Z","isPatch":true,"sender":{"key":"felipe.contreras@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8358?v=4"},"body":"On Tue, Oct 13, 2009 at 5:25 AM, Shawn O. Pearce <spearce@spearce.org> wrote:\n> The git-remote-curl backend detects if the remote server supports\n> the git-receive-pack service, and if so, runs git-send-pack in a\n> pipe to dump the command and pack data as a single POST request.\n>\n> The advertisements from the server that were obtained during the\n> discovery are passed into git-send-pack before the POST request\n> starts.  This permits git-send-pack to operate largely unmodified.\n>\n> For smaller packs (those under 1 MiB) a tranditional POST with a\n\nTraditional?\n\n> Content-Length is used, permitting interaction with any HTTP/1.0\n> compliant server.  The 1 MiB limit is arbitrary, but is sufficent\n> to fit most deltas created by human authors against text sources\n> with the occasional small binary file (e.g. few KiB icon image).\n>\n> For larger packs which cannot be spooled entirely into the\n> helper's memory space, the POST request requires HTTP/1.1 and\n> Transfer-Encoding: chunked.  This permits the client to upload an\n> unknown amount of data in one HTTP transaction without needing to\n> pregenerate the entire pack file locally.\n>\n> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>\n> CC: Daniel Barkalow <barkalow@iabervon.org>\n> ---\n>  Documentation/config.txt |    8 ++\n>  builtin-send-pack.c      |  116 ++++++++++++++++++++-\n>  remote-curl.c            |  258 +++++++++++++++++++++++++++++++++++++++++++++-\n>  send-pack.h              |    3 +-\n>  sideband.c               |   11 ++-\n>  transport.c              |    1 +\n>  6 files changed, 384 insertions(+), 13 deletions(-)\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index cd17814..7130d07 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1089,6 +1089,14 @@ http.maxRequests::\n>        How many HTTP requests to launch in parallel. Can be overridden\n>        by the 'GIT_HTTP_MAX_REQUESTS' environment variable. Default is 5.\n>\n> +http.postBuffer::\n> +       Maximum size in bytes of the buffer used by smart HTTP\n> +       transports when POSTing data to the remote system.\n> +       For requests larger than this buffer size, HTTP/1.1 and\n> +       Transfer-Encoding: chunked is used to avoid creating a\n> +       massive pack file locally.  Default is 1 MiB, which is\n> +       sufficient for most requests.\n> +\n>  http.lowSpeedLimit, http.lowSpeedTime::\n>        If the HTTP transfer speed is less than 'http.lowSpeedLimit'\n>        for longer than 'http.lowSpeedTime' seconds, the transfer is aborted.\n> diff --git a/builtin-send-pack.c b/builtin-send-pack.c\n> index 37e528e..654c3d4 100644\n> --- a/builtin-send-pack.c\n> +++ b/builtin-send-pack.c\n> @@ -2,9 +2,11 @@\n>  #include \"commit.h\"\n>  #include \"refs.h\"\n>  #include \"pkt-line.h\"\n> +#include \"sideband.h\"\n>  #include \"run-command.h\"\n>  #include \"remote.h\"\n>  #include \"send-pack.h\"\n> +#include \"quote.h\"\n>\n>  static const char send_pack_usage[] =\n>  \"git send-pack [--all | --mirror] [--dry-run] [--force] [--receive-pack=<git-receive-pack>] [--verbose] [--thin] [<host>:]<directory> [<ref>...]\\n\"\n> @@ -59,7 +61,7 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n>        memset(&po, 0, sizeof(po));\n>        po.argv = argv;\n>        po.in = -1;\n> -       po.out = fd;\n> +       po.out = args->one_shot_rpc ? -1 : fd;\n>        po.git_cmd = 1;\n>        if (start_command(&po))\n>                die_errno(\"git pack-objects failed\");\n> @@ -83,6 +85,20 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n>        }\n>\n>        close(po.in);\n> +\n> +       if (args->one_shot_rpc) {\n> +               char *buf = xmalloc(LARGE_PACKET_MAX);\n> +               while (1) {\n> +                       ssize_t n = xread(po.out, buf, LARGE_PACKET_MAX);\n> +                       if (n <= 0)\n> +                               break;\n> +                       send_sideband(fd, -1, buf, n, LARGE_PACKET_MAX);\n> +               }\n> +               free(buf);\n> +               close(po.out);\n> +               po.out = -1;\n> +       }\n> +\n>        if (finish_command(&po))\n>                return error(\"pack-objects died with strange error\");\n>        return 0;\n> @@ -303,6 +319,59 @@ static int refs_pushed(struct ref *ref)\n>        return 0;\n>  }\n>\n> +static void print_helper_status(struct ref *ref)\n> +{\n> +       struct strbuf buf = STRBUF_INIT;\n> +\n> +       for (; ref; ref = ref->next) {\n> +               const char *msg = NULL;\n> +               const char *res;\n> +\n> +               switch(ref->status) {\n> +               case REF_STATUS_NONE:\n> +                       res = \"error\";\n> +                       msg = \"no match\";\n> +                       break;\n> +\n> +               case REF_STATUS_OK:\n> +                       res = \"ok\";\n> +                       break;\n> +\n> +               case REF_STATUS_UPTODATE:\n> +                       res = \"ok\";\n> +                       msg = \"up to date\";\n> +                       break;\n> +\n> +               case REF_STATUS_REJECT_NONFASTFORWARD:\n> +                       res = \"error\";\n> +                       msg = \"non-fast forward\";\n> +                       break;\n> +\n> +               case REF_STATUS_REJECT_NODELETE:\n> +               case REF_STATUS_REMOTE_REJECT:\n> +                       res = \"error\";\n> +                       break;\n> +\n> +               case REF_STATUS_EXPECTING_REPORT:\n> +               default:\n> +                       continue;\n> +               }\n> +\n> +               strbuf_reset(&buf);\n> +               strbuf_addf(&buf, \"%s %s\", res, ref->name);\n> +               if (ref->remote_status)\n> +                       msg = ref->remote_status;\n> +               if (msg) {\n> +                       strbuf_addch(&buf, ' ');\n> +                       quote_two_c_style(&buf, \"\", msg, 0);\n> +               }\n> +               strbuf_addch(&buf, '\\n');\n> +\n> +               safe_write(1, buf.buf, buf.len);\n> +       }\n> +       strbuf_release(&buf);\n> +}\n> +\n>  int send_pack(struct send_pack_args *args,\n>              int fd[], struct child_process *conn,\n>              struct ref *remote_refs,\n> @@ -310,6 +379,7 @@ int send_pack(struct send_pack_args *args,\n>  {\n>        int in = fd[0];\n>        int out = fd[1];\n> +       struct strbuf req_buf = STRBUF_INIT;\n>        struct ref *ref;\n>        int new_refs;\n>        int ask_for_status_report = 0;\n> @@ -391,14 +461,14 @@ int send_pack(struct send_pack_args *args,\n>                        char *new_hex = sha1_to_hex(ref->new_sha1);\n>\n>                        if (ask_for_status_report) {\n> -                               packet_write(out, \"%s %s %s%c%s\",\n> +                               packet_buf_write(&req_buf, \"%s %s %s%c%s\",\n>                                        old_hex, new_hex, ref->name, 0,\n>                                        \"report-status\");\n>                                ask_for_status_report = 0;\n>                                expect_status_report = 1;\n>                        }\n>                        else\n> -                               packet_write(out, \"%s %s %s\",\n> +                               packet_buf_write(&req_buf, \"%s %s %s\",\n>                                        old_hex, new_hex, ref->name);\n>                }\n>                ref->status = expect_status_report ?\n> @@ -406,7 +476,17 @@ int send_pack(struct send_pack_args *args,\n>                        REF_STATUS_OK;\n>        }\n>\n> -       packet_flush(out);\n> +       if (args->one_shot_rpc) {\n> +               if (!args->dry_run) {\n> +                       packet_buf_flush(&req_buf);\n> +                       send_sideband(out, -1, req_buf.buf, req_buf.len, LARGE_PACKET_MAX);\n> +               }\n> +       } else {\n> +               safe_write(out, req_buf.buf, req_buf.len);\n> +               packet_flush(out);\n> +       }\n> +       strbuf_release(&req_buf);\n> +\n>        if (new_refs && !args->dry_run) {\n>                if (pack_objects(out, remote_refs, extra_have, args) < 0) {\n>                        for (ref = remote_refs; ref; ref = ref->next)\n> @@ -414,11 +494,15 @@ int send_pack(struct send_pack_args *args,\n>                        return -1;\n>                }\n>        }\n> +       if (args->one_shot_rpc && !args->dry_run)\n> +               packet_flush(out);\n>\n>        if (expect_status_report)\n>                ret = receive_status(in, remote_refs);\n>        else\n>                ret = 0;\n> +       if (args->one_shot_rpc)\n> +               packet_flush(out);\n>\n>        if (ret < 0)\n>                return ret;\n> @@ -478,6 +562,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n>        struct extra_have_objects extra_have;\n>        struct ref *remote_refs, *local_refs;\n>        int ret;\n> +       int helper_status = 0;\n>        int send_all = 0;\n>        const char *receivepack = \"git-receive-pack\";\n>        int flags;\n> @@ -523,6 +608,14 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n>                                args.use_thin_pack = 1;\n>                                continue;\n>                        }\n> +                       if (!strcmp(arg, \"--one-shot-rpc\")) {\n> +                               args.one_shot_rpc = 1;\n> +                               continue;\n> +                       }\n> +                       if (!strcmp(arg, \"--helper-status\")) {\n> +                               helper_status = 1;\n> +                               continue;\n> +                       }\n>                        usage(send_pack_usage);\n>                }\n>                if (!dest) {\n> @@ -551,7 +644,14 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n>                }\n>        }\n>\n> -       conn = git_connect(fd, dest, receivepack, args.verbose ? CONNECT_VERBOSE : 0);\n> +       if (args.one_shot_rpc) {\n> +               conn = NULL;\n> +               fd[0] = 0;\n> +               fd[1] = 1;\n> +       } else {\n> +               conn = git_connect(fd, dest, receivepack,\n> +                       args.verbose ? CONNECT_VERBOSE : 0);\n> +       }\n>\n>        memset(&extra_have, 0, sizeof(extra_have));\n>\n> @@ -575,12 +675,16 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n>\n>        ret = send_pack(&args, fd, conn, remote_refs, &extra_have);\n>\n> +       if (helper_status)\n> +               print_helper_status(remote_refs);\n> +\n>        close(fd[1]);\n>        close(fd[0]);\n>\n>        ret |= finish_connect(conn);\n>\n> -       print_push_status(dest, remote_refs);\n> +       if (!helper_status)\n> +               print_push_status(dest, remote_refs);\n>\n>        if (!args.dry_run && remote) {\n>                struct ref *ref;\n> diff --git a/remote-curl.c b/remote-curl.c\n> index 42fd06c..529df42 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -5,17 +5,33 @@\n>  #include \"walker.h\"\n>  #include \"http.h\"\n>  #include \"pkt-line.h\"\n> +#include \"sideband.h\"\n>  #include \"run-command.h\"\n>\n>\n> +static size_t post_buffer_size = 16 * LARGE_PACKET_MAX;\n>  static struct remote *remote;\n>  static const char *url;\n>  static struct walker *walker;\n>\n> +static int http_options(const char *var, const char *value, void *cb)\n> +{\n> +       if (!strcmp(\"http.postbuffer\", var)) {\n> +               post_buffer_size = git_config_int(var, value);\n> +               if (post_buffer_size < LARGE_PACKET_MAX)\n> +                       post_buffer_size = LARGE_PACKET_MAX;\n> +               return 0;\n> +       }\n> +\n> +       return 0;\n> +}\n> +\n>  static void init_walker(void)\n>  {\n> -       if (!walker)\n> +       if (!walker) {\n>                walker = get_http_walker(url, remote);\n> +               git_config(http_options, NULL);\n> +       }\n>  }\n>\n>  struct discovery {\n> @@ -200,6 +216,193 @@ static void output_refs(struct ref *refs)\n>        free_refs(refs);\n>  }\n>\n> +struct rpc_state {\n> +       const char *service_name;\n> +       char *service_url;\n> +       char *hdr_content_type;\n> +       char *hdr_accept;\n> +       char *buf;\n> +       size_t alloc;\n> +       size_t len;\n> +       size_t pos;\n> +       int in;\n> +       int out;\n> +       unsigned verbose : 1;\n> +};\n> +\n> +static size_t rpc_out(void *ptr, size_t eltsize,\n> +               size_t nmemb, void *buffer_)\n> +{\n> +       size_t max = eltsize * nmemb;\n> +       struct rpc_state *state = buffer_;\n> +       size_t avail = state->len - state->pos;\n> +\n> +       if (!avail) {\n> +               avail = packet_read_line(state->out, state->buf, state->alloc);\n> +               if (!avail)\n> +                       return 0;\n> +               state->pos = 0;\n> +               state->len = avail;\n> +       }\n> +\n> +       if (max < avail);\n> +               avail = max;\n> +       memcpy(ptr, state->buf + state->pos, avail);\n> +       state->pos += avail;\n> +       return avail;\n> +}\n> +\n> +static size_t rpc_in(const void *ptr, size_t eltsize,\n> +               size_t nmemb, void *buffer_)\n> +{\n> +       size_t size = eltsize * nmemb;\n> +       struct rpc_state *state = buffer_;\n> +       write_or_die(state->in, ptr, size);\n> +       return size;\n> +}\n> +\n> +static int post_rpc(struct rpc_state *state)\n> +{\n> +       struct active_request_slot *slot;\n> +       struct slot_results results;\n> +       struct curl_slist *headers = NULL;\n> +       int err = 0, large_request = 0;\n> +\n> +       /* Try to load the entire request, if we can fit it into the\n> +        * allocated buffer space we can use HTTP/1.0 and avoid the\n> +        * chunked encoding mess.\n> +        */\n> +       while (1) {\n> +               size_t left = state->alloc - state->len;\n> +               char *buf = state->buf + state->len;\n> +               int n;\n> +\n> +               if (left < LARGE_PACKET_MAX) {\n> +                       large_request = 1;\n> +                       break;\n> +               }\n> +\n> +               n = packet_read_line(state->out, buf, left);\n> +               if (!n)\n> +                       break;\n> +               state->len += n;\n> +       }\n> +\n> +       slot = get_active_slot();\n> +       slot->results = &results;\n> +\n> +       curl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n> +       curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n> +       curl_easy_setopt(slot->curl, CURLOPT_URL, state->service_url);\n> +\n> +       headers = curl_slist_append(headers, state->hdr_content_type);\n> +       headers = curl_slist_append(headers, state->hdr_accept);\n> +\n> +       if (large_request) {\n> +               /* The request body is large and the size cannot be predicted.\n> +                * We must use chunked encoding to send it.\n> +                */\n> +               headers = curl_slist_append(headers, \"Expect: 100-continue\");\n> +               headers = curl_slist_append(headers, \"Transfer-Encoding: chunked\");\n> +               curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);\n> +               curl_easy_setopt(slot->curl, CURLOPT_INFILE, state);\n> +               if (state->verbose) {\n> +                       fprintf(stderr, \"POST %s (chunked)\\n\", state->service_name);\n> +                       fflush(stderr);\n> +               }\n> +\n> +       } else {\n> +               /* We know the complete request size in advance, use the\n> +                * more normal Content-Length approach.\n> +                */\n> +               curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, state->buf);\n> +               curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, state->len);\n> +               if (state->verbose) {\n> +                       fprintf(stderr, \"POST %s (%lu bytes)\\n\",\n> +                               state->service_name, (unsigned long)state->len);\n> +                       fflush(stderr);\n> +               }\n> +       }\n> +\n> +       curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n> +       curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in);\n> +       curl_easy_setopt(slot->curl, CURLOPT_FILE, state);\n> +\n> +       if (start_active_slot(slot)) {\n> +               run_active_slot(slot);\n> +               if (results.curl_result != CURLE_OK) {\n> +                       err |= error(\"RPC failed; result=%d, HTTP code = %ld\",\n> +                               results.curl_result, results.http_code);\n> +               }\n> +       }\n> +       curl_slist_free_all(headers);\n> +       return err;\n> +}\n> +\n> +static int one_shot_rpc_service(const char *service,\n> +       int verbose,\n> +       const char **client_argv,\n> +       struct discovery *heads,\n> +       struct strbuf *result)\n> +{\n> +       struct child_process client;\n> +       struct rpc_state state;\n> +       struct strbuf buf = STRBUF_INIT;\n> +       int err = 0;\n> +\n> +       init_walker();\n> +       memset(&client, 0, sizeof(client));\n> +       client.in = -1;\n> +       client.out = -1;\n> +       client.git_cmd = 1;\n> +       client.argv = client_argv;\n> +       if (start_command(&client))\n> +               die(\"%s failed to execute\", client_argv[0]);\n> +       if (heads)\n> +               write_or_die(client.in, heads->buf, heads->len);\n> +\n> +       memset(&state, 0, sizeof(state));\n> +       state.alloc = post_buffer_size;\n> +       state.buf = xmalloc(state.alloc);\n> +       state.in = client.in;\n> +       state.out = client.out;\n> +       state.service_name = service;\n> +       state.verbose = !!verbose;\n> +\n> +       strbuf_addf(&buf, \"%s/%s\", url, service);\n> +       state.service_url = strbuf_detach(&buf, NULL);\n> +\n> +       strbuf_addf(&buf, \"Content-Type: application/x-%s-request\", service);\n> +       state.hdr_content_type = strbuf_detach(&buf, NULL);\n> +\n> +       strbuf_addf(&buf, \"Accept: application/x-%s-response\", service);\n> +       state.hdr_accept = strbuf_detach(&buf, NULL);\n> +\n> +       while (!err) {\n> +               int n = packet_read_line(state.out, state.buf, state.alloc);\n> +               if (!n)\n> +                       break;\n> +               state.pos = 0;\n> +               state.len = n;\n> +               err |= post_rpc(&state);\n> +       }\n> +       if (result)\n> +               strbuf_read(result, client.out, 0);\n> +\n> +       close(client.in);\n> +       close(client.out);\n> +       client.in = -1;\n> +       client.out = -1;\n> +\n> +       err |= finish_command(&client);\n> +       free(state.service_url);\n> +       free(state.hdr_content_type);\n> +       free(state.hdr_accept);\n> +       free(state.buf);\n> +       strbuf_release(&buf);\n> +       return err;\n> +}\n> +\n>  struct fetch_args {\n>        unsigned verbose : 1;\n>  };\n> @@ -289,7 +492,8 @@ static void parse_fetch(struct strbuf *buf, int multiple)\n>\n>  struct push_args {\n>        unsigned dry_run : 1,\n> -               verbose : 1;\n> +               verbose : 1,\n> +               thin : 1;\n>  };\n>\n>  static int push_dav(struct push_args *args, int nr_spec, char **specs)\n> @@ -314,6 +518,51 @@ static int push_dav(struct push_args *args, int nr_spec, char **specs)\n>        return 0;\n>  }\n>\n> +static int push_git(struct discovery *heads,\n> +       struct push_args *args, int nr_spec, char **specs)\n> +{\n> +       struct strbuf res = STRBUF_INIT;\n> +       const char **argv;\n> +       int argc = 0, i, err;\n> +\n> +       argv = xmalloc((10 + nr_spec) * sizeof(char*));\n> +       argv[argc++] = \"send-pack\";\n> +       argv[argc++] = \"--one-shot-rpc\";\n> +       argv[argc++] = \"--helper-status\";\n> +       if (args->thin)\n> +               argv[argc++] = \"--thin\";\n> +       if (args->dry_run)\n> +               argv[argc++] = \"--dry-run\";\n> +       if (args->verbose)\n> +               argv[argc++] = \"--verbose\";\n> +       argv[argc++] = url;\n> +       for (i = 0; i < nr_spec; i++)\n> +               argv[argc++] = specs[i];\n> +       argv[argc++] = NULL;\n> +\n> +       err = one_shot_rpc_service(\"git-receive-pack\",\n> +               args->verbose,\n> +               argv, heads, &res);\n> +       if (res.len)\n> +               safe_write(1, res.buf, res.len);\n> +       strbuf_release(&res);\n> +       free(argv);\n> +       return err;\n> +}\n> +\n> +static int push(struct push_args *args, int nr_spec, char **specs)\n> +{\n> +       struct discovery *heads = discover_refs(\"git-receive-pack\");\n> +       int ret;\n> +\n> +       if (heads->proto_git)\n> +               ret = push_git(heads, args, nr_spec, specs);\n> +       else\n> +               ret = push_dav(args, nr_spec, specs);\n> +       free_discovery(heads);\n> +       return ret;\n> +}\n> +\n>  static void parse_push(struct strbuf *buf)\n>  {\n>        char **specs = NULL;\n> @@ -330,6 +579,8 @@ static void parse_push(struct strbuf *buf)\n>                        args.dry_run = 1;\n>                else if (!strcmp(buf->buf, \"option verbose\"))\n>                        args.verbose = 1;\n> +               else if (!strcmp(buf->buf, \"option thin\"))\n> +                       args.thin = 1;\n>                else\n>                        die(\"http transport does not support %s\", buf->buf);\n>\n> @@ -340,7 +591,7 @@ static void parse_push(struct strbuf *buf)\n>                        break;\n>        } while (1);\n>\n> -       if (push_dav(&args, nr_spec, specs))\n> +       if (push(&args, nr_spec, specs))\n>                exit(128); /* error already reported */\n>        for (i = 0; i < nr_spec; i++)\n>                free(specs[i]);\n> @@ -395,6 +646,7 @@ int main(int argc, const char **argv)\n>                        printf(\"fetch-multiple\\n\");\n>                        printf(\"push\\n\");\n>                        printf(\"option dry-run\\n\");\n> +                       printf(\"option thin\\n\");\n>                        printf(\"option verbose\\n\");\n>                        printf(\"\\n\");\n>                        fflush(stdout);\n> diff --git a/send-pack.h b/send-pack.h\n> index 8b3cf02..a7f4abf 100644\n> --- a/send-pack.h\n> +++ b/send-pack.h\n> @@ -8,7 +8,8 @@ struct send_pack_args {\n>                force_update:1,\n>                use_thin_pack:1,\n>                use_ofs_delta:1,\n> -               dry_run:1;\n> +               dry_run:1,\n> +               one_shot_rpc:1;\n>  };\n>\n>  int send_pack(struct send_pack_args *args,\n> diff --git a/sideband.c b/sideband.c\n> index 899b1ff..d5ffa1c 100644\n> --- a/sideband.c\n> +++ b/sideband.c\n> @@ -135,9 +135,14 @@ ssize_t send_sideband(int fd, int band, const char *data, ssize_t sz, int packet\n>                n = sz;\n>                if (packet_max - 5 < n)\n>                        n = packet_max - 5;\n> -               sprintf(hdr, \"%04x\", n + 5);\n> -               hdr[4] = band;\n> -               safe_write(fd, hdr, 5);\n> +               if (0 <= band) {\n> +                       sprintf(hdr, \"%04x\", n + 5);\n> +                       hdr[4] = band;\n> +                       safe_write(fd, hdr, 5);\n> +               } else {\n> +                       sprintf(hdr, \"%04x\", n + 4);\n> +                       safe_write(fd, hdr, 4);\n> +               }\n>                safe_write(fd, p, n);\n>                p += n;\n>                sz -= n;\n> diff --git a/transport.c b/transport.c\n> index 6d9652d..2ff1650 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -731,6 +731,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n>                                 NULL);\n>        }\n>\n> +       memset(&args, 0, sizeof(args));\n>        args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n>        args.force_update = !!(flags & TRANSPORT_PUSH_FORCE);\n>        args.use_thin_pack = data->thin;\n> --\n> 1.6.5.52.g0ff2e\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n\n\n\n-- \nFelipe Contreras\n"},{"id":"124855","messageId":"alpine.DEB.1.00.0910132003520.4985@pacific.mpi-cbg.de","threadId":"21220","inReplyTo":"20091013044141.GA18961@glandium.org","subject":"Re: [RFC PATCH v2 09/16] Move WebDAV HTTP push under remote-curl","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-10-13T18:04:35Z","receivedAt":"2009-10-13T18:04:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 13 Oct 2009, Mike Hommey wrote:\n\n> On Mon, Oct 12, 2009 at 07:25:08PM -0700, Shawn O. Pearce wrote:\n> > The remote helper interface now supports the push capability, which \n> > can be used to ask the implementation to push one or more specs to the \n> > remote repository.  For remote-curl we implement this by calling the \n> > existing WebDAV based git-http-push executable.\n> > \n> > Internally the helper interface uses the push_refs transport hook so \n> > that the complexity of the refspec parsing and matching can be reused \n> > between remote implementations.  When possible however the helper \n> > protocol uses source ref name rather than the source SHA-1, thereby \n> > allowing the helper to access this name if it is useful.\n> \n> It's been a while I haven't followed changes in the remote code but this \n> looks nice, though I haven't checked thoroughly. I guess the next step \n> would be to kill http-push as an external program.\n\nIlari made signs on IRC that he got something working.  After, ahem, \nsaying a few things about the design of the remote helpers.\n\nCiao,\nDscho\n"},{"id":"124856","messageId":"20091013180552.GW9261@spearce.org","threadId":"21220","inReplyTo":"alpine.LNX.2.00.0910122326130.32515@iabervon.org","subject":"Re: [RFC PATCH v2 07/16] remote-helpers: Fetch more than one ref in a batch","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T18:05:52Z","receivedAt":"2009-10-13T18:05:52Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> wrote:\n> On Mon, 12 Oct 2009, Shawn O. Pearce wrote:\n> > Some network protocols (e.g. native git://) are able to fetch more\n> > than one ref at a time and reduce the overall transfer cost by\n> > combining the requests into a single exchange.  Instead of feeding\n> > each fetch request one at a time to the helper, feed all of them\n> > at once so the helper can decide whether or not it should batch them.\n> > \n> > Because 'fetch' was already released in 1.6.5 we introduce the new\n> > fetch-multiple capability/command to signal that the helper wants\n> > to use batch oriented approach to fetching refs.\n> \n> In 1.6.5, there's no way to call a helper other than git-remote-curl, and \n> no way to call git-remote-curl unless 1.6.5 was built with it. So I think \n> the protocol is not set in stone quite yet. It's documentated for being an \n> API, and it's supposed to be that, but it's not quite there in this \n> version.\n\nI originally had this change as just redefining \"fetch\" to be batch\noriented and requiring a blank line to terminate the batch group,\nbut reconsidered when 1.6.5 shipped and I realized this code was\nin 1.6.5.\n\nBut, yea, you are right, with no way to invoke anything except\nremote-curl there really isn't that much of a problem if we change\nthe protocol.\n\n-- \nShawn.\n"},{"id":"124861","messageId":"20091013181041.GX9261@spearce.org","threadId":"21220","inReplyTo":"4AD42C52.80205@viscovery.net","subject":"Re: [RFC PATCH v2 01/16] pkt-line: Add strbuf based functions","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T18:10:41Z","receivedAt":"2009-10-13T18:10:41Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Sixt <j.sixt@viscovery.net> wrote:\n> Shawn O. Pearce schrieb:\n> > -int packet_read_line(int fd, char *buffer, unsigned size)\n> > +static int packet_length(unsigned *ret_len, const char *linelen)\n...\n> > +\t*ret_len = len;\n> > +\treturn 0;\n> > +}\n> \n> len can be signed: Valid lengths fit into a signed int. Then you can\n> 'return len;' on success and 'return -1;' on failure and don't need return\n> the result by reference. packet_read_line() ultimately converts it to int\n> anyway:\n\nGreat catch, thanks.  This is actually from a prior version of code\nwhere I was exposing this function to callers... but even then the\nmethod could have just returned int with the value because as you\npoint out, all valid lengths fit in int and must be >= 0.\n \n-- \nShawn.\n"},{"id":"124863","messageId":"20091013182427.GA9261@spearce.org","threadId":"21220","inReplyTo":"4AD42C87.6000205@viscovery.net","subject":"Re: [RFC PATCH v2 12/16] Smart fetch and push over HTTP: server side","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T18:24:27Z","receivedAt":"2009-10-13T18:24:27Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Sixt <j.sixt@viscovery.net> wrote:\n> Shawn O. Pearce schrieb:\n> > diff --git a/http-backend.c b/http-backend.c\n> \n> #include \"run-command.h\" is missing here because you added it already in\n> patch 10/16 unnecessarily.\n> \n> > +\tif (start_command(&cld))\n> > +\t\tdie_errno(\"Cannot start %s\", argv[0]);\n> > ...\n> > +\tif (finish_command(&cld))\n> > +\t\tdie(\"%s terminated with error\", argv[0]);\n> \n> start_command and finish_command already write an error message for you\n> that includes argv[0] and errno. You can just exit(1) here.\n\nWhoops on both; thanks for the catch.\n\n-- \nShawn.\n"},{"id":"124866","messageId":"20091013184531.GB9261@spearce.org","threadId":"21220","inReplyTo":"alpine.LNX.2.00.0910122357230.32515@iabervon.org","subject":"Re: [RFC PATCH v2 08/16] remote-helpers: Support custom transport options","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T18:45:31Z","receivedAt":"2009-10-13T18:45:31Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> wrote:\n> > diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\n> > +'option' <name>::\n> > +\tThis helper supports the option <name> under fetch-multiple.\n> > +\n> \n> I'm a bit surprised that the options only apply in a fetch-multiple \n> section, rather than getting set at the beginning and applying to \n> everything for that run. At least, I think an \"option\" command should be \n> useable outside of a fetch-multiple (or possible future grouping \n> construct) and have global scope.\n\nIn hindsight, I agree with you.\n\nI'll respin the series so the set_option method in the transport\nforwards the options immediately to the helper and lets the helper\ndecide whether it accepts or rejects the option string.  This will\nclean up the capabilities interface since we no longer need to dump\nthe list of options we support in the helper, and as you point out,\nit will make a lot more sense to just set the options for this\ntransport instance.\n \n> >  REF LIST ATTRIBUTES\n> >  -------------------\n> >  \n> > @@ -76,10 +80,26 @@ None are defined yet, but the caller must accept any which are supplied.\n> >  \n> >  FETCH OPTIONS\n> >  -------------\n> > +To enable an option the helper must list it in 'capabilities'.\n> >  \n> >  'option verbose'::\n> >  \tPrint more verbose activity messages to stderr.\n> \n> I think you mis-split the above part; your previoud patch declared this \n> option without declaring any way to use it. Might be worth allowing \n> multiple \"verboses\" and \"quiet\" or \"option verbosity quiet\"/\"option \n> verbosity verbose verbose\".\n\nHmmph.  \"option verbosity verbose verbose\" is a bit verbose, don't\nyou think?  :-)\n\nI think we should just forward the verbosity setting from the\nfrontend: \"option verbosity [0-n]\" where n is the number of\ntimes -v appeared on the command line/how verbose the user wants.\n \n> > +'option uploadpack' <command>::\n> > +\tThe program to use on the remote side to generate a pack.\n> \n> I sort of feel like the helper ought to read this one out of the config \n> file itself if it wants it.\n\nEh, true, but you can also set this on the command line.  An open\nquestion I still have for myself is how to set this in HTTP\ntransports.\n\nThe reason why I care is Gerrit Code Review has overloaded the\n'git-receive-pack' executable and taught it more command line flags:\n\n  $ ssh r git receive-pack -h\n  git receive-pack PROJECT.git [--cc EMAIL ...] [--help (-h)] [--reviewer (--re) EMAIL ...]\n\n   PROJECT.git             : project name\n   --cc EMAIL              : CC user on change(s)\n   --help (-h)             : display this help text\n   --reviewer (--re) EMAIL : request reviewer for change(s)\n\nWhich is typically invoked as:\n\n  git push --receive-pack \"git-receive-pack --reviewer spearce@spearce.org\" URL REFSPEC\n\nFolks actually have scripts which make this invocation for them, so\nthey can insert in the proper reviewer and/or cc arguments.  Since\nthe arguments vary its hard to set this up in the configuration file.\n\nOver SSH this is fine, we obtain the arguments off the SSH command\nline string and its no big deal.  Over git:// this would fail as\ngit-daemon can't parse the line anymore.  Over HTTP this also is not\ngoing to work since the service can't receive arbitrary arguements.\n\nMy primary motivator for doing smart HTTP now is folks who are\nstuck behind firewalls that permit only HTTP through their local\nproxy servers are unable to communicate with a Gerrit Code Review\ninstance over SSH on port 29418.  That --reviewer flag above is a\nvery useful feature of Gerrit that I somehow have to support for\nthe HTTP transport too.\n\nI started down the road of currying this data into the backend by\nat least exposing the option to the helper.  How the helper reads\nand uses it is up to the helper.\n\nBut given that the value can come in from the command line or from\nthe configuration file, I think this should be handled by fetch\nor push porcelain and fed through the helper protocol, and not be\nsomething that the helper reads from the config directly.\n\n> In general, it would be good to have \n> transport.c and remote.c out of the business of knowing this sort of \n> protocol-specific (albiet specific now to two protocols) information. (Of \n> course, the native protocol's transport methods are in transport.c, so \n> that's there, but I'd like to move that to a transport-native.c someday.)\n\nAgreed, but I have no solution for you due to the --receive-pack\nand --upload-pack arguments supported by the command line git push\nand git fetch/pull porcelain.\n\nBut I have been trying to extend the helper interface in a way\nthat would allow us to eject the native transport code entirely\ninto a helper.  We may never bother, there are some advantages to\nbeing in the push/fetch client process, but I also didn't want to\nget stuck in a corner.\n\nI think with my series we do almost everything we need to support\nnative git:// in an external helper process rather than builtin.\nWe honor the pack lock file system used by fetch to maintain safe\nconcurrent mutations.  We use push_refs API and signal back the\ncomplete information from the remote side.  We permit arbitrary\nmessage strings per ref to be returned by the helper.  Etc.\n \n> > +'option followtags'::\n> > +\tAggressively fetch annotated tags if possible.\n> \n> I assume this means to fetch tags which annotate objects we have or are \n> fetching? (As opposed to fetching any annotated tag we could possibly \n> fetch, even if we don't otherwise care about the tag or the thing it \n> tags.) It's obvious in the context of git's config options, but I'd like \n> this document to avoid assuming that context, and the option could apply \n> more generally.\n\nYes.  I'll extend the documentation further in the next iteration.\n \n> > +'option thin'::\n> > +\tTransfer the data as a thin pack if possible.\n> \n> Does anyone still use non-default thinness? \n\nIts a command line option on the porcelain.  Until we remove\nthe command line flag I think we should still try to honor it\nin implementations that understand that notion.\n\n-- \nShawn.\n"},{"id":"124890","messageId":"alpine.LNX.2.00.0910131550170.32515@iabervon.org","threadId":"21220","inReplyTo":"20091013184531.GB9261@spearce.org","subject":"Re: [RFC PATCH v2 08/16] remote-helpers: Support custom transport options","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-10-13T20:39:17Z","receivedAt":"2009-10-13T20:39:17Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 13 Oct 2009, Shawn O. Pearce wrote:\n\n> Daniel Barkalow <barkalow@iabervon.org> wrote:\n> > > diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\n> > > +'option' <name>::\n> > > +\tThis helper supports the option <name> under fetch-multiple.\n> > > +\n> > \n> > I'm a bit surprised that the options only apply in a fetch-multiple \n> > section, rather than getting set at the beginning and applying to \n> > everything for that run. At least, I think an \"option\" command should be \n> > useable outside of a fetch-multiple (or possible future grouping \n> > construct) and have global scope.\n> \n> In hindsight, I agree with you.\n> \n> I'll respin the series so the set_option method in the transport\n> forwards the options immediately to the helper and lets the helper\n> decide whether it accepts or rejects the option string.  This will\n> clean up the capabilities interface since we no longer need to dump\n> the list of options we support in the helper, and as you point out,\n> it will make a lot more sense to just set the options for this\n> transport instance.\n\nGreat. That also makes the \"fetch-multiple\" start flag no longer \nstrictly necessary.\n\n> > >  REF LIST ATTRIBUTES\n> > >  -------------------\n> > >  \n> > > @@ -76,10 +80,26 @@ None are defined yet, but the caller must accept any which are supplied.\n> > >  \n> > >  FETCH OPTIONS\n> > >  -------------\n> > > +To enable an option the helper must list it in 'capabilities'.\n> > >  \n> > >  'option verbose'::\n> > >  \tPrint more verbose activity messages to stderr.\n> > \n> > I think you mis-split the above part; your previoud patch declared this \n> > option without declaring any way to use it. Might be worth allowing \n> > multiple \"verboses\" and \"quiet\" or \"option verbosity quiet\"/\"option \n> > verbosity verbose verbose\".\n> \n> Hmmph.  \"option verbosity verbose verbose\" is a bit verbose, don't\n> you think?  :-)\n> \n> I think we should just forward the verbosity setting from the\n> frontend: \"option verbosity [0-n]\" where n is the number of\n> times -v appeared on the command line/how verbose the user wants.\n\nI think it can go down to -1. Also, I remember needing to have a \"-v\" \nget eaten by the fetch/pull itself, so that you can have fetch/pull tell \nyou stuff without having the actual protocol interaction getting more \nverbose, so it's not exactly the same numbers that get through.\n\n> > > +'option uploadpack' <command>::\n> > > +\tThe program to use on the remote side to generate a pack.\n> > \n> > I sort of feel like the helper ought to read this one out of the config \n> > file itself if it wants it.\n> \n> Eh, true, but you can also set this on the command line.  An open\n> question I still have for myself is how to set this in HTTP\n> transports.\n\nGood point. I doubt people really want to change the executable name for \nthe same remote between different runs, though.\n\n> The reason why I care is Gerrit Code Review has overloaded the\n> 'git-receive-pack' executable and taught it more command line flags:\n> \n>   $ ssh r git receive-pack -h\n>   git receive-pack PROJECT.git [--cc EMAIL ...] [--help (-h)] [--reviewer (--re) EMAIL ...]\n> \n>    PROJECT.git             : project name\n>    --cc EMAIL              : CC user on change(s)\n>    --help (-h)             : display this help text\n>    --reviewer (--re) EMAIL : request reviewer for change(s)\n> \n> Which is typically invoked as:\n> \n>   git push --receive-pack \"git-receive-pack --reviewer spearce@spearce.org\" URL REFSPEC\n\nIt seems to me like what we really want is a channel to communicate extra \ninformation to hooks on the server side (or a modified command on the \nserver). Like:\n\ngit push --recv-opt \"--reviewer spearce@spearce.org\"\n\nor something like that. This would arrange to pass that option to whatever \nserver-side code is responsible for accepting pushes, without specifying \nhow it gets there.\n\n> Folks actually have scripts which make this invocation for them, so\n> they can insert in the proper reviewer and/or cc arguments.  Since\n> the arguments vary its hard to set this up in the configuration file.\n> \n> Over SSH this is fine, we obtain the arguments off the SSH command\n> line string and its no big deal.  Over git:// this would fail as\n> git-daemon can't parse the line anymore.  Over HTTP this also is not\n> going to work since the service can't receive arbitrary arguements.\n> \n> My primary motivator for doing smart HTTP now is folks who are\n> stuck behind firewalls that permit only HTTP through their local\n> proxy servers are unable to communicate with a Gerrit Code Review\n> instance over SSH on port 29418.  That --reviewer flag above is a\n> very useful feature of Gerrit that I somehow have to support for\n> the HTTP transport too.\n> \n> I started down the road of currying this data into the backend by\n> at least exposing the option to the helper.  How the helper reads\n> and uses it is up to the helper.\n> \n> But given that the value can come in from the command line or from\n> the configuration file, I think this should be handled by fetch\n> or push porcelain and fed through the helper protocol, and not be\n> something that the helper reads from the config directly.\n\nI agree that the porcelain and transport.c should handle \"--reviewer \nspearce@spearce.org\", but I still think they shouldn't handle the\n\"git-receive-pack\" part, and it would probably be easier on everyone if it \nwas separated in the porcelain, and the native transport knew to stick it \ninto the receive-pack command line.\n\n> > In general, it would be good to have \n> > transport.c and remote.c out of the business of knowing this sort of \n> > protocol-specific (albiet specific now to two protocols) information. (Of \n> > course, the native protocol's transport methods are in transport.c, so \n> > that's there, but I'd like to move that to a transport-native.c someday.)\n> \n> Agreed, but I have no solution for you due to the --receive-pack\n> and --upload-pack arguments supported by the command line git push\n> and git fetch/pull porcelain.\n> \n> But I have been trying to extend the helper interface in a way\n> that would allow us to eject the native transport code entirely\n> into a helper.  We may never bother, there are some advantages to\n> being in the push/fetch client process, but I also didn't want to\n> get stuck in a corner.\n> \n> I think with my series we do almost everything we need to support\n> native git:// in an external helper process rather than builtin.\n> We honor the pack lock file system used by fetch to maintain safe\n> concurrent mutations.  We use push_refs API and signal back the\n> complete information from the remote side.  We permit arbitrary\n> message strings per ref to be returned by the helper.  Etc.\n\nI think it would be a worthwhile exercise to actually write the series to \neject all of the transports into helpers. Then I think we should probably \nmake the extensions to the helper code this requires, eject the rsync one, \nand put bundles and native in contrib instead of using them.\n\n> > > +'option followtags'::\n> > > +\tAggressively fetch annotated tags if possible.\n> > \n> > I assume this means to fetch tags which annotate objects we have or are \n> > fetching? (As opposed to fetching any annotated tag we could possibly \n> > fetch, even if we don't otherwise care about the tag or the thing it \n> > tags.) It's obvious in the context of git's config options, but I'd like \n> > this document to avoid assuming that context, and the option could apply \n> > more generally.\n> \n> Yes.  I'll extend the documentation further in the next iteration.\n>  \n> > > +'option thin'::\n> > > +\tTransfer the data as a thin pack if possible.\n> > \n> > Does anyone still use non-default thinness? \n> \n> Its a command line option on the porcelain.  Until we remove\n> the command line flag I think we should still try to honor it\n> in implementations that understand that notion.\n\nActually, the command line supports turning it on, and it defaults to on. \nSo I think your helper can safely assume that it's on. :)\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"124891","messageId":"20091013205258.GD9261@spearce.org","threadId":"21220","inReplyTo":"alpine.LNX.2.00.0910131550170.32515@iabervon.org","subject":"Re: [RFC PATCH v2 08/16] remote-helpers: Support custom transport options","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T20:52:58Z","receivedAt":"2009-10-13T20:52:58Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> wrote:\n> On Tue, 13 Oct 2009, Shawn O. Pearce wrote:\n> > > > +'option thin'::\n> > > > +\tTransfer the data as a thin pack if possible.\n> > > \n> > > Does anyone still use non-default thinness? \n> > \n> > Its a command line option on the porcelain.\n> \n> Actually, the command line supports turning it on, and it defaults to on. \n> So I think your helper can safely assume that it's on. :)\n\nFor fetch it defaults to \"on\", but for push I think it defaults\nto \"off\".  Turning it on when pushing on a low bandwidth network\nconnection might actually be useful to an end-user.\n\n-- \nShawn.\n"},{"id":"124901","messageId":"m3ocobf067.fsf@localhost.localdomain","threadId":"21220","inReplyTo":"1255400715-10508-6-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v2 05/16] Add multi_ack_2 capability to fetch-pack/upload-pack","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-10-13T21:35:07Z","receivedAt":"2009-10-13T21:35:07Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Shawn O. Pearce\" <spearce@spearce.org> writes:\n\n> When multi_ack_2 is enabled the ACK continue messages returned by the\n> remote upload-pack are broken out to describe the different states\n> within the peer.  This permits the client to better understand the\n> server's in-memory state.\n\nErrr... can't you find better name than multi_ack_2?  Perhaps\nmulti_ack_detailed or something...\n\n> \n> The fetch-pack/upload-pack protocol now looks like:\n[...]\n\n> ACK %s continue\n> -----------------------------------\n>   * multi_ack only:\n> \n>     Sent in response to \"have\".\n> \n>     The remote side wants the client to consider this object as\n>     common, and immediately stop transmitting additional \"have\"\n>     lines for objects that are reachable from it.  The reason\n>     the client should stop is not given, but is one of the two\n>     cases below available under multi_ack_2.\n> \n> ACK %s common\n> -----------------------------------\n>   * multi_ack_2 only:\n> \n>     Sent in response to \"have\".  Both sides have this object.\n>     Like with \"ACK %s continue\" above the client should stop\n>     sending have lines reachable for objects from the argument.\n> \n> ACK %s ready\n> -----------------------------------\n>   * multi_ack_2 only:\n> \n>     Sent in response to \"have\".\n> \n>     The client should stop transmitting objects which are reachable\n>     from the argument, and send \"done\" soon to get the objects.\n> \n>     If the remote side has the specified object, it should\n>     first send an \"ACK %s common\" message prior to sending\n>     \"ACK %s ready\".\n> \n>     Clients may still submit additional \"have\" lines if there are\n>     more side branches for the client to explore that might be added\n>     to the common set and reduce the number of objects to transfer.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"124902","messageId":"20091013213632.GF9261@spearce.org","threadId":"21220","inReplyTo":"m3ocobf067.fsf@localhost.localdomain","subject":"Re: [RFC PATCH v2 05/16] Add multi_ack_2 capability to fetch-pack/upload-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T21:36:32Z","receivedAt":"2009-10-13T21:36:32Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> wrote:\n> \"Shawn O. Pearce\" <spearce@spearce.org> writes:\n> \n> > When multi_ack_2 is enabled the ACK continue messages returned by the\n> > remote upload-pack are broken out to describe the different states\n> > within the peer.  This permits the client to better understand the\n> > server's in-memory state.\n> \n> Errr... can't you find better name than multi_ack_2?  Perhaps\n> multi_ack_detailed or something...\n\nUh... yes.  Good idea.  Thanks.  :-)\n \n-- \nShawn.\n"},{"id":"124903","messageId":"alpine.LNX.2.00.0910131732260.32515@iabervon.org","threadId":"21220","inReplyTo":"20091013205258.GD9261@spearce.org","subject":"Re: [RFC PATCH v2 08/16] remote-helpers: Support custom transport options","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-10-13T21:41:48Z","receivedAt":"2009-10-13T21:41:48Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 13 Oct 2009, Shawn O. Pearce wrote:\n\n> Daniel Barkalow <barkalow@iabervon.org> wrote:\n> > On Tue, 13 Oct 2009, Shawn O. Pearce wrote:\n> > > > > +'option thin'::\n> > > > > +\tTransfer the data as a thin pack if possible.\n> > > > \n> > > > Does anyone still use non-default thinness? \n> > > \n> > > Its a command line option on the porcelain.\n> > \n> > Actually, the command line supports turning it on, and it defaults to on. \n> > So I think your helper can safely assume that it's on. :)\n> \n> For fetch it defaults to \"on\", but for push I think it defaults\n> to \"off\".  Turning it on when pushing on a low bandwidth network\n> connection might actually be useful to an end-user.\n\nNope, on ~line 849 of transport.c, it gets set for all native-transport \nhandlers, and never gets turned off. Looks like a misconversion 2 years \nago defaulting \"data->thin\" to 1 instead of 0, but it seems not to have \ncaused problems.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"124904","messageId":"20091013215131.GG9261@spearce.org","threadId":"21220","inReplyTo":"alpine.LNX.2.00.0910131732260.32515@iabervon.org","subject":"Re: [RFC PATCH v2 08/16] remote-helpers: Support custom transport options","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-13T21:51:31Z","receivedAt":"2009-10-13T21:51:31Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> wrote:\n> On Tue, 13 Oct 2009, Shawn O. Pearce wrote:\n> > \n> > For fetch it defaults to \"on\", but for push I think it defaults\n> > to \"off\".\n> \n> Nope, on ~line 849 of transport.c, it gets set for all native-transport \n> handlers, and never gets turned off. Looks like a misconversion 2 years \n> ago defaulting \"data->thin\" to 1 instead of 0, but it seems not to have \n> caused problems.\n\nThe rationale for why it was off by default on push was to avoid\nmaking suboptimal packs on the server due to needing to complete\nthe thin pack out.  If its been off for 2 years I guess its not\nthat big of a deal.  We can probably leave thin out then and just\nstart deprecating the option.\n\n\nActually... right now I'm more concerned about why a negotiation\nover the native protocol is making the client mention \"have v1.5.5\"\n(yes, the tag object) when I'm trying to fetch between two git.git\nrepositories that are only 1 Junio work day apart.\n\nIn the grand scheme of things over TCP or SSH, that's a few extra\nhave lines.  Over the stateless HTTP its why the final request is\nbloating out to 8 KiB worth of have lines.  In one Junio work day\nwe should only be seeing a few hundred new objects, yea pu rewinds,\nbut v1.5.5 isn't the best common base available to us... IMHO we\nshouldn't have even mentioned it.\n\nBut fixing this is independent of smart HTTP, I'll try to circle\nback later and look at why I'm seeing this oddity in the common\ncommit negotiation.\n\n-- \nShawn.\n"}]}