{"thread":{"id":"21411","subject":"[RFC PATCH v4 05/26] Move \"get_ack()\" back to fetch-pack","startedAt":"2009-10-29T00:00:22Z","lastAt":"2009-10-31T00:13:14Z","messageCount":60,"participants":["Shawn O. Pearce","Clemens Buchacher","Junio C Hamano","Jakub Narebski","Tay Ray Chuan","Jeff King","David Brown"],"isPatch":true,"patchVersion":4,"patchTotal":26},"messages":[{"id":"126193","messageId":"1256774448-7625-1-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":null,"subject":"[RFC PATCH v4 00/26] Return of smart HTTP","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:22Z","receivedAt":"2009-10-29T00:00:22Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"I think this is the final spin of the smart HTTP series. I've\ncollected patches from a few others (thanks folks!) and added\ntests specific to the smart variant of the HTTP transport.\n\nAt this point, I think it is \"next ready\"... but would appreciate\nany additional feedback if folks identify something we should\naddress before hitting next.\n\n\nClemens Buchacher (1):\n remote-helpers: return successfully if everything up-to-date\n\nMark Lodato (5):\n http-backend: add GIT_PROJECT_ROOT environment var\n http-backend: reword some documentation\n http-backend: use mod_alias instead of mod_rewrite\n http-backend: add example for gitweb on same URL\n http-backend: more explict LocationMatch\n\nShawn O. Pearce (18):\n pkt-line: Add strbuf based functions\n pkt-line: Make packet_read_line easier to debug\n fetch-pack: Use a strbuf to compose the want list\n Move \"get_ack()\" back to fetch-pack\n Add multi_ack_detailed capability to fetch-pack/upload-pack\n remote-curl: Refactor walker initialization\n fetch: Allow transport -v -v -v to set verbosity to 3\n remote-helpers: Fetch more than one ref in a batch\n remote-helpers: Support custom transport options\n Move WebDAV HTTP push under remote-curl\n Git-aware CGI to provide dumb HTTP transport\n Add stateless RPC options to upload-pack, receive-pack\n Smart fetch and push over HTTP: server side\n Discover refs via smart HTTP server when available\n Smart push over HTTP: client side\n Smart fetch over HTTP: client side\n Smart HTTP fetch: gzip requests\n test smart http fetch and push\n\nTay Ray Chuan (2):\n http-push: fix check condition on http.c::finish_http_pack_request()\n t5540-http-push: remove redundant fetches\n\n .gitignore | 1 +\n Documentation/config.txt | 8 +\n Documentation/git-http-backend.txt | 170 ++++++++\n Documentation/git-remote-helpers.txt | 85 ++++-\n Makefile | 1 +\n builtin-fetch-pack.c | 210 ++++++++--\n builtin-fetch.c | 2 +-\n builtin-receive-pack.c | 26 +-\n builtin-send-pack.c | 116 +++++-\n cache.h | 1 -\n commit.c | 10 +-\n commit.h | 2 +-\n connect.c | 21 -\n fetch-pack.h | 3 +-\n http-backend.c | 627 ++++++++++++++++++++++++++++\n http-push.c | 31 ++-\n http.c | 13 +-\n http.h | 2 +\n pkt-line.c | 86 ++++-\n pkt-line.h | 4 +\n remote-curl.c | 759 ++++++++++++++++++++++++++++++++--\n send-pack.h | 3 +-\n sideband.c | 11 +-\n t/lib-httpd/apache.conf | 20 +\n t/t5540-http-push.sh | 18 +-\n t/t5541-http-push.sh | 103 +++++\n t/t5550-http-fetch.sh | 8 +-\n t/t5551-http-fetch.sh | 87 ++++\n transport-helper.c | 264 +++++++++++-\n transport.c | 32 +--\n transport.h | 2 +-\n upload-pack.c | 71 +++-\n 32 files changed, 2574 insertions(+), 223 deletions(-)\n create mode 100644 Documentation/git-http-backend.txt\n create mode 100644 http-backend.c\n create mode 100755 t/t5541-http-push.sh\n create mode 100755 t/t5551-http-fetch.sh\n"},{"id":"126214","messageId":"1256774448-7625-2-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 01/26] http-push: fix check condition on http.c::finish_http_pack_request()","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:23Z","receivedAt":"2009-10-29T00:00:23Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"From: Tay Ray Chuan \n\nCheck that http.c::finish_http_pack_request() returns 0 (for success).\n\nSigned-off-by: Tay Ray Chuan \nSigned-off-by: Shawn O. Pearce \n---\n http-push.c | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/http-push.c b/http-push.c\nindex 00e83dc..cc5d4b8 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -604,7 +604,7 @@ static void finish_request(struct transfer_request *request)\n \t\t\tpreq = (struct http_pack_request *)request->userData;\n \n \t\t\tif (preq) {\n-\t\t\t\tif (finish_http_pack_request(preq) > 0)\n+\t\t\t\tif (finish_http_pack_request(preq) == 0)\n \t\t\t\t\tfail = 0;\n \t\t\t\trelease_http_pack_request(preq);\n \t\t\t}\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126217","messageId":"1256774448-7625-3-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 02/26] pkt-line: Add strbuf based functions","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:24Z","receivedAt":"2009-10-29T00:00:24Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"These routines help to work with pkt-line values inside of a strbuf,\npermitting simple formatting of buffered network messages.\n\nSigned-off-by: Shawn O. Pearce \n---\n pkt-line.c | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++--------\n pkt-line.h | 4 +++\n 2 files changed, 76 insertions(+), 12 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex b691abe..bd603f8 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -42,17 +42,19 @@ void packet_flush(int fd)\n \tsafe_write(fd, \"0000\", 4);\n }\n \n+void packet_buf_flush(struct strbuf *buf)\n+{\n+\tstrbuf_add(buf, \"0000\", 4);\n+}\n+\n #define hex(a) (hexchar[(a) & 15])\n-void packet_write(int fd, const char *fmt, ...)\n+static char buffer[1000];\n+static unsigned format_packet(const char *fmt, va_list args)\n {\n-\tstatic char buffer[1000];\n \tstatic char hexchar[] = \"0123456789abcdef\";\n-\tva_list args;\n \tunsigned n;\n \n-\tva_start(args, fmt);\n \tn = vsnprintf(buffer + 4, sizeof(buffer) - 4, fmt, args);\n-\tva_end(args);\n \tif (n >= sizeof(buffer)-4)\n \t\tdie(\"protocol error: impossibly long line\");\n \tn += 4;\n@@ -60,9 +62,31 @@ void packet_write(int fd, const char *fmt, ...)\n \tbuffer[1] = hex(n >> 8);\n \tbuffer[2] = hex(n >> 4);\n \tbuffer[3] = hex(n);\n+\treturn n;\n+}\n+\n+void packet_write(int fd, const char *fmt, ...)\n+{\n+\tva_list args;\n+\tunsigned n;\n+\n+\tva_start(args, fmt);\n+\tn = format_packet(fmt, args);\n+\tva_end(args);\n \tsafe_write(fd, buffer, n);\n }\n \n+void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n+{\n+\tva_list args;\n+\tunsigned n;\n+\n+\tva_start(args, fmt);\n+\tn = format_packet(fmt, args);\n+\tva_end(args);\n+\tstrbuf_add(buf, buffer, n);\n+}\n+\n static void safe_read(int fd, void *buffer, unsigned size)\n {\n \tssize_t ret = read_in_full(fd, buffer, size);\n@@ -72,15 +96,11 @@ static void safe_read(int fd, void *buffer, unsigned size)\n \t\tdie(\"The remote end hung up unexpectedly\");\n }\n \n-int packet_read_line(int fd, char *buffer, unsigned size)\n+static int packet_length(const char *linelen)\n {\n \tint n;\n-\tunsigned len;\n-\tchar linelen[4];\n-\n-\tsafe_read(fd, linelen, 4);\n+\tint len = 0;\n \n-\tlen = 0;\n \tfor (n = 0; n < 4; n++) {\n \t\tunsigned char c = linelen[n];\n \t\tlen <<= 4;\n@@ -96,8 +116,20 @@ int packet_read_line(int fd, char *buffer, unsigned size)\n \t\t\tlen += c - 'A' + 10;\n \t\t\tcontinue;\n \t\t}\n-\t\tdie(\"protocol error: bad line length character\");\n+\t\treturn -1;\n \t}\n+\treturn len;\n+}\n+\n+int packet_read_line(int fd, char *buffer, unsigned size)\n+{\n+\tint len;\n+\tchar linelen[4];\n+\n+\tsafe_read(fd, linelen, 4);\n+\tlen = packet_length(linelen);\n+\tif (len < 0)\n+\t\tdie(\"protocol error: bad line length character\");\n \tif (!len)\n \t\treturn 0;\n \tlen -= 4;\n@@ -107,3 +139,31 @@ int packet_read_line(int fd, char *buffer, unsigned size)\n \tbuffer[len] = 0;\n \treturn len;\n }\n+\n+int packet_get_line(struct strbuf *out,\n+\tchar **src_buf, size_t *src_len)\n+{\n+\tint len;\n+\n+\tif (*src_len < 4)\n+\t\treturn -1;\n+\tlen = packet_length(*src_buf);\n+\tif (len < 0)\n+\t\treturn -1;\n+\tif (!len) {\n+\t\t*src_buf += 4;\n+\t\t*src_len -= 4;\n+\t\treturn 0;\n+\t}\n+\tif (*src_len < len)\n+\t\treturn -2;\n+\n+\t*src_buf += 4;\n+\t*src_len -= 4;\n+\tlen -= 4;\n+\n+\tstrbuf_add(out, *src_buf, len);\n+\t*src_buf += len;\n+\t*src_len -= len;\n+\treturn len;\n+}\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 9df653f..1e5dcfe 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -2,14 +2,18 @@\n #define PKTLINE_H\n \n #include \"git-compat-util.h\"\n+#include \"strbuf.h\"\n \n /*\n * Silly packetized line writing interface\n */\n void packet_flush(int fd);\n void packet_write(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n+void packet_buf_flush(struct strbuf *buf);\n+void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n \n int packet_read_line(int fd, char *buffer, unsigned size);\n+int packet_get_line(struct strbuf *out, char **src_buf, size_t *src_len);\n ssize_t safe_write(int, const void *, ssize_t);\n \n #endif\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126194","messageId":"1256774448-7625-4-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:25Z","receivedAt":"2009-10-29T00:00:25Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When there is an error parsing the 4 byte length component we now\nNUL terminate the string and display it as part of the die message,\nthis may hint as to what data was misunderstood by the application.\n\nSigned-off-by: Shawn O. Pearce \n---\n pkt-line.c | 8 +++++---\n 1 files changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex bd603f8..893dd3c 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -124,12 +124,14 @@ static int packet_length(const char *linelen)\n int packet_read_line(int fd, char *buffer, unsigned size)\n {\n \tint len;\n-\tchar linelen[4];\n+\tchar linelen[5];\n \n \tsafe_read(fd, linelen, 4);\n \tlen = packet_length(linelen);\n-\tif (len < 0)\n-\t\tdie(\"protocol error: bad line length character\");\n+\tif (len < 0) {\n+\t\tlinelen[4] = '\\0';\n+\t\tdie(\"protocol error: bad line length character: %s\", linelen);\n+\t}\n \tif (!len)\n \t\treturn 0;\n \tlen -= 4;\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126218","messageId":"1256774448-7625-5-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 04/26] fetch-pack: Use a strbuf to compose the want list","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:26Z","receivedAt":"2009-10-29T00:00:26Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This change is being offered as a refactoring to make later\ncommits in the smart HTTP series easier.\n\nBy changing the enabled capabilities to be formatted in a strbuf\nit is easier to add a new capability to the set of supported\ncapabilities.\n\nBy formatting the want portion of the request into a strbuf and\nwriting it as a whole block we can later decide to hold onto\nthe req_buf (instead of releasing it) to recycle in stateless\ncommunications.\n\nSigned-off-by: Shawn O. Pearce \n---\n builtin-fetch-pack.c | 52 ++++++++++++++++++++++++++++++++-----------------\n commit.c | 10 +++-----\n commit.h | 2 +-\n 3 files changed, 39 insertions(+), 25 deletions(-)\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex 629735f..783c2b0 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -165,6 +165,7 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tconst unsigned char *sha1;\n \tunsigned in_vain = 0;\n \tint got_continue = 0;\n+\tstruct strbuf req_buf = STRBUF_INIT;\n \n \tif (marked)\n \t\tfor_each_ref(clear_marks, NULL);\n@@ -175,6 +176,7 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\n \t\tunsigned char *remote = refs->old_sha1;\n+\t\tconst char *remote_hex;\n \t\tstruct object *o;\n \n \t\t/*\n@@ -192,27 +194,36 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tif (!fetching)\n-\t\t\tpacket_write(fd[1], \"want %s%s%s%s%s%s%s%s\\n\",\n-\t\t\t\t sha1_to_hex(remote),\n-\t\t\t\t (multi_ack ? \" multi_ack\" : \"\"),\n-\t\t\t\t (use_sideband == 2 ? \" side-band-64k\" : \"\"),\n-\t\t\t\t (use_sideband == 1 ? \" side-band\" : \"\"),\n-\t\t\t\t (args.use_thin_pack ? \" thin-pack\" : \"\"),\n-\t\t\t\t (args.no_progress ? \" no-progress\" : \"\"),\n-\t\t\t\t (args.include_tag ? \" include-tag\" : \"\"),\n-\t\t\t\t (prefer_ofs_delta ? \" ofs-delta\" : \"\"));\n-\t\telse\n-\t\t\tpacket_write(fd[1], \"want %s\\n\", sha1_to_hex(remote));\n+\t\tremote_hex = sha1_to_hex(remote);\n+\t\tif (!fetching) {\n+\t\t\tstruct strbuf c = STRBUF_INIT;\n+\t\t\tif (multi_ack) strbuf_addstr(&c, \" multi_ack\");\n+\t\t\tif (use_sideband == 2) strbuf_addstr(&c, \" side-band-64k\");\n+\t\t\tif (use_sideband == 1) strbuf_addstr(&c, \" side-band\");\n+\t\t\tif (args.use_thin_pack) strbuf_addstr(&c, \" thin-pack\");\n+\t\t\tif (args.no_progress) strbuf_addstr(&c, \" no-progress\");\n+\t\t\tif (args.include_tag) strbuf_addstr(&c, \" include-tag\");\n+\t\t\tif (prefer_ofs_delta) strbuf_addstr(&c, \" ofs-delta\");\n+\t\t\tpacket_buf_write(&req_buf, \"want %s%s\\n\", remote_hex, c.buf);\n+\t\t\tstrbuf_release(&c);\n+\t\t} else\n+\t\t\tpacket_buf_write(&req_buf, \"want %s\\n\", remote_hex);\n \t\tfetching++;\n \t}\n+\n+\tif (!fetching) {\n+\t\tstrbuf_release(&req_buf);\n+\t\tpacket_flush(fd[1]);\n+\t\treturn 1;\n+\t}\n+\n \tif (is_repository_shallow())\n-\t\twrite_shallow_commits(fd[1], 1);\n+\t\twrite_shallow_commits(&req_buf, 1);\n \tif (args.depth > 0)\n-\t\tpacket_write(fd[1], \"deepen %d\", args.depth);\n-\tpacket_flush(fd[1]);\n-\tif (!fetching)\n-\t\treturn 1;\n+\t\tpacket_buf_write(&req_buf, \"deepen %d\", args.depth);\n+\tpacket_buf_flush(&req_buf);\n+\n+\tsafe_write(fd[1], req_buf.buf, req_buf.len);\n \n \tif (args.depth > 0) {\n \t\tchar line[1024];\n@@ -296,6 +307,8 @@ done:\n \t\tmulti_ack = 0;\n \t\tflushes++;\n \t}\n+\tstrbuf_release(&req_buf);\n+\n \twhile (flushes || multi_ack) {\n \t\tint ack = get_ack(fd[0], result_sha1);\n \t\tif (ack) {\n@@ -809,6 +822,7 @@ struct ref *fetch_pack(struct fetch_pack_args *my_args,\n \n \tif (args.depth > 0) {\n \t\tstruct cache_time mtime;\n+\t\tstruct strbuf sb = STRBUF_INIT;\n \t\tchar *shallow = git_path(\"shallow\");\n \t\tint fd;\n \n@@ -826,12 +840,14 @@ struct ref *fetch_pack(struct fetch_pack_args *my_args,\n \n \t\tfd = hold_lock_file_for_update(&lock, shallow,\n \t\t\t\t\t LOCK_DIE_ON_ERROR);\n-\t\tif (!write_shallow_commits(fd, 0)) {\n+\t\tif (!write_shallow_commits(&sb, 0)\n+\t\t || write_in_full(fd, sb.buf, sb.len) != sb.len) {\n \t\t\tunlink_or_warn(shallow);\n \t\t\trollback_lock_file(&lock);\n \t\t} else {\n \t\t\tcommit_lock_file(&lock);\n \t\t}\n+\t\tstrbuf_release(&sb);\n \t}\n \n \treprepare_packed_git();\ndiff --git a/commit.c b/commit.c\nindex fedbd5e..471efb0 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -199,7 +199,7 @@ struct commit_graft *lookup_commit_graft(const unsigned char *sha1)\n \treturn commit_graft[pos];\n }\n \n-int write_shallow_commits(int fd, int use_pack_protocol)\n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n {\n \tint i, count = 0;\n \tfor (i = 0; i < commit_graft_nr; i++)\n@@ -208,12 +208,10 @@ int write_shallow_commits(int fd, int use_pack_protocol)\n \t\t\t\tsha1_to_hex(commit_graft[i]->sha1);\n \t\t\tcount++;\n \t\t\tif (use_pack_protocol)\n-\t\t\t\tpacket_write(fd, \"shallow %s\", hex);\n+\t\t\t\tpacket_buf_write(out, \"shallow %s\", hex);\n \t\t\telse {\n-\t\t\t\tif (write_in_full(fd, hex, 40) != 40)\n-\t\t\t\t\tbreak;\n-\t\t\t\tif (write_str_in_full(fd, \"\\n\") != 1)\n-\t\t\t\t\tbreak;\n+\t\t\t\tstrbuf_addstr(out, hex);\n+\t\t\t\tstrbuf_addch(out, '\\n');\n \t\t\t}\n \t\t}\n \treturn count;\ndiff --git a/commit.h b/commit.h\nindex f4fc5c5..817c75c 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -131,7 +131,7 @@ extern struct commit_list *get_octopus_merge_bases(struct commit_list *in);\n \n extern int register_shallow(const unsigned char *sha1);\n extern int unregister_shallow(const unsigned char *sha1);\n-extern int write_shallow_commits(int fd, int use_pack_protocol);\n+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n extern int is_repository_shallow(void);\n extern struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\tint depth, int shallow_flag, int not_shallow_flag);\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126192","messageId":"1256774448-7625-6-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 05/26] Move \"get_ack()\" back to fetch-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:27Z","receivedAt":"2009-10-29T00:00:27Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"In 41cb7488 Linus moved this function to connect.c for reuse inside\nof the git-clone-pack command. That was 2005, but in 2006 Junio\nretired git-clone-pack in commit efc7fa53. Since then the only\ncaller has been fetch-pack. Since this ACK/NAK exchange is only\nused by the fetch-pack/upload-pack protocol we should keep move\nit back to a private detail of fetch-pack.\n\nSigned-off-by: Shawn O. Pearce \n---\n builtin-fetch-pack.c | 21 +++++++++++++++++++++\n cache.h | 1 -\n connect.c | 21 ---------------------\n 3 files changed, 21 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex 783c2b0..7c09d46 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -157,6 +157,27 @@ static const unsigned char *get_rev(void)\n \treturn commit->object.sha1;\n }\n \n+static int get_ack(int fd, unsigned char *result_sha1)\n+{\n+\tstatic char line[1000];\n+\tint len = packet_read_line(fd, line, sizeof(line));\n+\n+\tif (!len)\n+\t\tdie(\"git fetch-pack: expected ACK/NAK, got EOF\");\n+\tif (line[len-1] == '\\n')\n+\t\tline[--len] = 0;\n+\tif (!strcmp(line, \"NAK\"))\n+\t\treturn 0;\n+\tif (!prefixcmp(line, \"ACK \")) {\n+\t\tif (!get_sha1_hex(line+4, result_sha1)) {\n+\t\t\tif (strstr(line+45, \"continue\"))\n+\t\t\t\treturn 2;\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\tdie(\"git fetch_pack: expected ACK/NAK, got '%s'\", line);\n+}\n+\n static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t struct ref *refs)\n {\ndiff --git a/cache.h b/cache.h\nindex a5eeead..4e283be 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -856,7 +856,6 @@ extern struct ref *find_ref_by_name(const struct ref *list, const char *name);\n extern struct child_process *git_connect(int fd[2], const char *url, const char *prog, int flags);\n extern int finish_connect(struct child_process *conn);\n extern int path_match(const char *path, int nr, char **match);\n-extern int get_ack(int fd, unsigned char *result_sha1);\n struct extra_have_objects {\n \tint nr, alloc;\n \tunsigned char (*array)[20];\ndiff --git a/connect.c b/connect.c\nindex 7945e38..839a103 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -107,27 +107,6 @@ int server_supports(const char *feature)\n \t\tstrstr(server_capabilities, feature) != NULL;\n }\n \n-int get_ack(int fd, unsigned char *result_sha1)\n-{\n-\tstatic char line[1000];\n-\tint len = packet_read_line(fd, line, sizeof(line));\n-\n-\tif (!len)\n-\t\tdie(\"git fetch-pack: expected ACK/NAK, got EOF\");\n-\tif (line[len-1] == '\\n')\n-\t\tline[--len] = 0;\n-\tif (!strcmp(line, \"NAK\"))\n-\t\treturn 0;\n-\tif (!prefixcmp(line, \"ACK \")) {\n-\t\tif (!get_sha1_hex(line+4, result_sha1)) {\n-\t\t\tif (strstr(line+45, \"continue\"))\n-\t\t\t\treturn 2;\n-\t\t\treturn 1;\n-\t\t}\n-\t}\n-\tdie(\"git fetch_pack: expected ACK/NAK, got '%s'\", line);\n-}\n-\n int path_match(const char *path, int nr, char **match)\n {\n \tint i;\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126195","messageId":"1256774448-7625-7-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 06/26] Add multi_ack_detailed capability to fetch-pack/upload-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:28Z","receivedAt":"2009-10-29T00:00:28Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When multi_ack_detailed is enabled the ACK continue messages returned\nby the remote upload-pack are broken out to describe the different\nstates within the peer. This permits the client to better understand\nthe server's in-memory state.\n\nThe fetch-pack/upload-pack protocol now looks like:\n\nNAK\n---------------------------------\n Always sent in response to \"done\" if there was no common base\n selected from the \"have\" lines (or no have lines were sent).\n\n * no multi_ack or multi_ack_detailed:\n\n Sent when the client has sent a pkt-line flush (\"0000\") and\n the server has not yet found a common base object.\n\n * either multi_ack or multi_ack_detailed:\n\n Always sent in response to a pkt-line flush.\n\nACK %s\n-----------------------------------\n * no multi_ack or multi_ack_detailed:\n\n Sent in response to \"have\" when the object exists on the remote\n side and is therefore an object in common between the peers.\n The argument is the SHA-1 of the common object.\n\n * either multi_ack or multi_ack_detailed:\n\n Sent in response to \"done\" if there are common objects.\n The argument is the last SHA-1 determined to be common.\n\nACK %s continue\n-----------------------------------\n * multi_ack only:\n\n Sent in response to \"have\".\n\n The remote side wants the client to consider this object as\n common, and immediately stop transmitting additional \"have\"\n lines for objects that are reachable from it. The reason\n the client should stop is not given, but is one of the two\n cases below available under multi_ack_detailed.\n\nACK %s common\n-----------------------------------\n * multi_ack_detailed only:\n\n Sent in response to \"have\". Both sides have this object.\n Like with \"ACK %s continue\" above the client should stop\n sending have lines reachable for objects from the argument.\n\nACK %s ready\n-----------------------------------\n * multi_ack_detailed only:\n\n Sent in response to \"have\".\n\n The client should stop transmitting objects which are reachable\n from the argument, and send \"done\" soon to get the objects.\n\n If the remote side has the specified object, it should\n first send an \"ACK %s common\" message prior to sending\n \"ACK %s ready\".\n\n Clients may still submit additional \"have\" lines if there are\n more side branches for the client to explore that might be added\n to the common set and reduce the number of objects to transfer.\n\nSigned-off-by: Shawn O. Pearce \n---\n builtin-fetch-pack.c | 41 ++++++++++++++++++++++++++++++++---------\n upload-pack.c | 31 ++++++++++++++++++-------------\n 2 files changed, 50 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex 7c09d46..615f549 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -157,7 +157,15 @@ static const unsigned char *get_rev(void)\n \treturn commit->object.sha1;\n }\n \n-static int get_ack(int fd, unsigned char *result_sha1)\n+enum ack_type {\n+\tNAK = 0,\n+\tACK,\n+\tACK_continue,\n+\tACK_common,\n+\tACK_ready\n+};\n+\n+static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n {\n \tstatic char line[1000];\n \tint len = packet_read_line(fd, line, sizeof(line));\n@@ -167,12 +175,16 @@ static int get_ack(int fd, unsigned char *result_sha1)\n \tif (line[len-1] == '\\n')\n \t\tline[--len] = 0;\n \tif (!strcmp(line, \"NAK\"))\n-\t\treturn 0;\n+\t\treturn NAK;\n \tif (!prefixcmp(line, \"ACK \")) {\n \t\tif (!get_sha1_hex(line+4, result_sha1)) {\n \t\t\tif (strstr(line+45, \"continue\"))\n-\t\t\t\treturn 2;\n-\t\t\treturn 1;\n+\t\t\t\treturn ACK_continue;\n+\t\t\tif (strstr(line+45, \"common\"))\n+\t\t\t\treturn ACK_common;\n+\t\t\tif (strstr(line+45, \"ready\"))\n+\t\t\t\treturn ACK_ready;\n+\t\t\treturn ACK;\n \t\t}\n \t}\n \tdie(\"git fetch_pack: expected ACK/NAK, got '%s'\", line);\n@@ -218,7 +230,8 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\tremote_hex = sha1_to_hex(remote);\n \t\tif (!fetching) {\n \t\t\tstruct strbuf c = STRBUF_INIT;\n-\t\t\tif (multi_ack) strbuf_addstr(&c, \" multi_ack\");\n+\t\t\tif (multi_ack == 2) strbuf_addstr(&c, \" multi_ack_detailed\");\n+\t\t\tif (multi_ack == 1) strbuf_addstr(&c, \" multi_ack\");\n \t\t\tif (use_sideband == 2) strbuf_addstr(&c, \" side-band-64k\");\n \t\t\tif (use_sideband == 1) strbuf_addstr(&c, \" side-band\");\n \t\t\tif (args.use_thin_pack) strbuf_addstr(&c, \" thin-pack\");\n@@ -298,18 +311,23 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t\t\tif (args.verbose && ack)\n \t\t\t\t\tfprintf(stderr, \"got ack %d %s\\n\", ack,\n \t\t\t\t\t\t\tsha1_to_hex(result_sha1));\n-\t\t\t\tif (ack == 1) {\n+\t\t\t\tswitch (ack) {\n+\t\t\t\tcase ACK:\n \t\t\t\t\tflushes = 0;\n \t\t\t\t\tmulti_ack = 0;\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgoto done;\n-\t\t\t\t} else if (ack == 2) {\n+\t\t\t\tcase ACK_common:\n+\t\t\t\tcase ACK_ready:\n+\t\t\t\tcase ACK_continue: {\n \t\t\t\t\tstruct commit *commit =\n \t\t\t\t\t\tlookup_commit(result_sha1);\n \t\t\t\t\tmark_common(commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tin_vain = 0;\n \t\t\t\t\tgot_continue = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t\t}\n \t\t\t\t}\n \t\t\t} while (ack);\n \t\t\tflushes--;\n@@ -336,7 +354,7 @@ done:\n \t\t\tif (args.verbose)\n \t\t\t\tfprintf(stderr, \"got ack (%d) %s\\n\", ack,\n \t\t\t\t\tsha1_to_hex(result_sha1));\n-\t\t\tif (ack == 1)\n+\t\t\tif (ack == ACK)\n \t\t\t\treturn 0;\n \t\t\tmulti_ack = 1;\n \t\t\tcontinue;\n@@ -618,7 +636,12 @@ static struct ref *do_fetch_pack(int fd[2],\n \n \tif (is_repository_shallow() && !server_supports(\"shallow\"))\n \t\tdie(\"Server does not support shallow clients\");\n-\tif (server_supports(\"multi_ack\")) {\n+\tif (server_supports(\"multi_ack_detailed\")) {\n+\t\tif (args.verbose)\n+\t\t\tfprintf(stderr, \"Server supports multi_ack_detailed\\n\");\n+\t\tmulti_ack = 2;\n+\t}\n+\telse if (server_supports(\"multi_ack\")) {\n \t\tif (args.verbose)\n \t\t\tfprintf(stderr, \"Server supports multi_ack\\n\");\n \t\tmulti_ack = 1;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 38ddac2..f1dc3a3 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -498,7 +498,7 @@ static int get_common_commits(void)\n {\n \tstatic char line[1000];\n \tunsigned char sha1[20];\n-\tchar hex[41], last_hex[41];\n+\tchar last_hex[41];\n \n \tsave_commit_buffer = 0;\n \n@@ -515,19 +515,22 @@ static int get_common_commits(void)\n \t\tif (!prefixcmp(line, \"have \")) {\n \t\t\tswitch (got_sha1(line+5, sha1)) {\n \t\t\tcase -1: /* they have what we do not */\n-\t\t\t\tif (multi_ack && ok_to_give_up())\n-\t\t\t\t\tpacket_write(1, \"ACK %s continue\\n\",\n-\t\t\t\t\t\t sha1_to_hex(sha1));\n+\t\t\t\tif (multi_ack && ok_to_give_up()) {\n+\t\t\t\t\tconst char *hex = sha1_to_hex(sha1);\n+\t\t\t\t\tif (multi_ack == 2)\n+\t\t\t\t\t\tpacket_write(1, \"ACK %s ready\\n\", hex);\n+\t\t\t\t\telse\n+\t\t\t\t\t\tpacket_write(1, \"ACK %s continue\\n\", hex);\n+\t\t\t\t}\n \t\t\t\tbreak;\n \t\t\tdefault:\n-\t\t\t\tmemcpy(hex, sha1_to_hex(sha1), 41);\n-\t\t\t\tif (multi_ack) {\n-\t\t\t\t\tconst char *msg = \"ACK %s continue\\n\";\n-\t\t\t\t\tpacket_write(1, msg, hex);\n-\t\t\t\t\tmemcpy(last_hex, hex, 41);\n-\t\t\t\t}\n+\t\t\t\tmemcpy(last_hex, sha1_to_hex(sha1), 41);\n+\t\t\t\tif (multi_ack == 2)\n+\t\t\t\t\tpacket_write(1, \"ACK %s common\\n\", last_hex);\n+\t\t\t\telse if (multi_ack)\n+\t\t\t\t\tpacket_write(1, \"ACK %s continue\\n\", last_hex);\n \t\t\t\telse if (have_obj.nr == 1)\n-\t\t\t\t\tpacket_write(1, \"ACK %s\\n\", hex);\n+\t\t\t\t\tpacket_write(1, \"ACK %s\\n\", last_hex);\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tcontinue;\n@@ -587,7 +590,9 @@ static void receive_needs(void)\n \t\t get_sha1_hex(line+5, sha1_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n \t\t\t \"expected to get sha, not '%s'\", line);\n-\t\tif (strstr(line+45, \"multi_ack\"))\n+\t\tif (strstr(line+45, \"multi_ack_detailed\"))\n+\t\t\tmulti_ack = 2;\n+\t\telse if (strstr(line+45, \"multi_ack\"))\n \t\t\tmulti_ack = 1;\n \t\tif (strstr(line+45, \"thin-pack\"))\n \t\t\tuse_thin_pack = 1;\n@@ -681,7 +686,7 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow no-progress\"\n-\t\t\" include-tag\";\n+\t\t\" include-tag multi_ack_detailed\";\n \tstruct object *o = parse_object(sha1);\n \n \tif (!o)\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126216","messageId":"1256774448-7625-8-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 07/26] remote-curl: Refactor walker initialization","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:29Z","receivedAt":"2009-10-29T00:00:29Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"We will need the walker, url and remote in other functions as the\ncode grows larger to support smart HTTP. Extract this out into a\nset of globals we can easily reference once configured.\n\nSigned-off-by: Shawn O. Pearce \nCC: Daniel Barkalow \n---\n remote-curl.c | 24 ++++++++++++++----------\n 1 files changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 2faf1c6..478f3ea 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -5,7 +5,17 @@\n #include \"http.h\"\n #include \"exec_cmd.h\"\n \n-static struct ref *get_refs(struct walker *walker, const char *url)\n+static struct remote *remote;\n+static const char *url;\n+static struct walker *walker;\n+\n+static void init_walker(void)\n+{\n+\tif (!walker)\n+\t\twalker = get_http_walker(url, remote);\n+}\n+\n+static struct ref *get_refs(void)\n {\n \tstruct strbuf buffer = STRBUF_INIT;\n \tchar *data, *start, *mid;\n@@ -21,6 +31,7 @@ static struct ref *get_refs(struct walker *walker, const char *url)\n \trefs_url = xmalloc(strlen(url) + 11);\n \tsprintf(refs_url, \"%s/info/refs\", url);\n \n+\tinit_walker();\n \thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n \tswitch (http_ret) {\n \tcase HTTP_OK:\n@@ -78,10 +89,7 @@ static struct ref *get_refs(struct walker *walker, const char *url)\n \n int main(int argc, const char **argv)\n {\n-\tstruct remote *remote;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tconst char *url;\n-\tstruct walker *walker = NULL;\n \n \tgit_extract_argv0_path(argv[0]);\n \tsetup_git_directory();\n@@ -103,8 +111,7 @@ int main(int argc, const char **argv)\n \t\t\tbreak;\n \t\tif (!prefixcmp(buf.buf, \"fetch \")) {\n \t\t\tchar *obj = buf.buf + strlen(\"fetch \");\n-\t\t\tif (!walker)\n-\t\t\t\twalker = get_http_walker(url, remote);\n+\t\t\tinit_walker();\n \t\t\twalker->get_all = 1;\n \t\t\twalker->get_tree = 1;\n \t\t\twalker->get_history = 1;\n@@ -115,11 +122,8 @@ int main(int argc, const char **argv)\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n \t\t} else if (!strcmp(buf.buf, \"list\")) {\n-\t\t\tstruct ref *refs;\n+\t\t\tstruct ref *refs = get_refs();\n \t\t\tstruct ref *posn;\n-\t\t\tif (!walker)\n-\t\t\t\twalker = get_http_walker(url, remote);\n-\t\t\trefs = get_refs(walker, url);\n \t\t\tfor (posn = refs; posn; posn = posn->next) {\n \t\t\t\tif (posn->symref)\n \t\t\t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126212","messageId":"1256774448-7625-9-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 08/26] fetch: Allow transport -v -v -v to set verbosity to 3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:30Z","receivedAt":"2009-10-29T00:00:30Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Helpers might want a higher level of verbosity than just +1 (the\nporcelain default setting) and +2 (-v -v). Expand the field to\nallow verbosity in the range -1..3.\n\nSigned-off-by: Shawn O. Pearce \nCC: Daniel Barkalow \n---\n builtin-fetch.c | 2 +-\n transport.h | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin-fetch.c b/builtin-fetch.c\nindex cb48c57..52a9a42 100644\n--- a/builtin-fetch.c\n+++ b/builtin-fetch.c\n@@ -665,7 +665,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \n \ttransport = transport_get(remote, remote->url[0]);\n \tif (verbosity >= 2)\n-\t\ttransport->verbose = 1;\n+\t\ttransport->verbose = verbosity <= 3 ? verbosity : 3;\n \tif (verbosity < 0)\n \t\ttransport->verbose = -1;\n \tif (upload_pack)\ndiff --git a/transport.h b/transport.h\nindex c14da6f..e4e6177 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -25,7 +25,7 @@ struct transport {\n \n \tint (*disconnect)(struct transport *connection);\n \tchar *pack_lockfile;\n-\tsigned verbose : 2;\n+\tsigned verbose : 3;\n \t/* Force progress even if the output is not a tty */\n \tunsigned progress : 1;\n };\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126215","messageId":"1256774448-7625-10-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 09/26] remote-helpers: Fetch more than one ref in a batch","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:31Z","receivedAt":"2009-10-29T00:00:31Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Some network protocols (e.g. native git://) are able to fetch more\nthan one ref at a time and reduce the overall transfer cost by\ncombining the requests into a single exchange. Instead of feeding\neach fetch request one at a time to the helper, feed all of them\nat once so the helper can decide whether or not it should batch them.\n\nSigned-off-by: Shawn O. Pearce \nCC: Daniel Barkalow \n---\n Documentation/git-remote-helpers.txt | 14 ++++--\n remote-curl.c | 88 +++++++++++++++++++++++++++++----\n transport-helper.c | 39 +++++++++++----\n 3 files changed, 115 insertions(+), 26 deletions(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex 173ee23..e44d821 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -36,10 +36,16 @@ Commands are given by the caller on the helper's standard input, one per line.\n \tcomplete list, outputs a blank line.\n \n 'fetch' ::\n-\tFetches the given object, writing the necessary objects to the\n-\tdatabase. Outputs a blank line when the fetch is\n-\tcomplete. Only objects which were reported in the ref list\n-\twith a sha1 may be fetched this way.\n+\tFetches the given object, writing the necessary objects\n+\tto the database. Fetch commands are sent in a batch, one\n+\tper line, and the batch is terminated with a blank line.\n+\tOutputs a single blank line when all fetch commands in the\n+\tsame batch are complete. Only objects which were reported\n+\tin the ref list with a sha1 may be fetched this way.\n++\n+Optionally may output a 'lock ' line indicating a file under\n+GIT_DIR/objects/pack which is keeping a pack until refs can be\n+suitably updated.\n +\n Supported if the helper has the \"fetch\" capability.\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 478f3ea..22cd5c5 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -87,6 +87,81 @@ static struct ref *get_refs(void)\n \treturn refs;\n }\n \n+static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n+{\n+\tchar **targets = xmalloc(nr_heads * sizeof(char*));\n+\tint ret, i;\n+\n+\tfor (i = 0; i < nr_heads; i++)\n+\t\ttargets[i] = xstrdup(sha1_to_hex(to_fetch[i]->old_sha1));\n+\n+\tinit_walker();\n+\twalker->get_all = 1;\n+\twalker->get_tree = 1;\n+\twalker->get_history = 1;\n+\twalker->get_verbosely = 0;\n+\twalker->get_recover = 0;\n+\tret = walker_fetch(walker, nr_heads, targets, NULL, NULL);\n+\n+\tfor (i = 0; i < nr_heads; i++)\n+\t\tfree(targets[i]);\n+\tfree(targets);\n+\n+\treturn ret ? error(\"Fetch failed.\") : 0;\n+}\n+\n+static void parse_fetch(struct strbuf *buf)\n+{\n+\tstruct ref **to_fetch = NULL;\n+\tstruct ref *list_head = NULL;\n+\tstruct ref **list = &list_head;\n+\tint alloc_heads = 0, nr_heads = 0;\n+\n+\tdo {\n+\t\tif (!prefixcmp(buf->buf, \"fetch \")) {\n+\t\t\tchar *p = buf->buf + strlen(\"fetch \");\n+\t\t\tchar *name;\n+\t\t\tstruct ref *ref;\n+\t\t\tunsigned char old_sha1[20];\n+\n+\t\t\tif (strlen(p) < 40 || get_sha1_hex(p, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected sha/ref, got %s'\", p);\n+\t\t\tif (p[40] == ' ')\n+\t\t\t\tname = p + 41;\n+\t\t\telse if (!p[40])\n+\t\t\t\tname = \"\";\n+\t\t\telse\n+\t\t\t\tdie(\"protocol error: expected sha/ref, got %s'\", p);\n+\n+\t\t\tref = alloc_ref(name);\n+\t\t\thashcpy(ref->old_sha1, old_sha1);\n+\n+\t\t\t*list = ref;\n+\t\t\tlist = &ref->next;\n+\n+\t\t\tALLOC_GROW(to_fetch, nr_heads + 1, alloc_heads);\n+\t\t\tto_fetch[nr_heads++] = ref;\n+\t\t}\n+\t\telse\n+\t\t\tdie(\"http transport does not support %s\", buf->buf);\n+\n+\t\tstrbuf_reset(buf);\n+\t\tif (strbuf_getline(buf, stdin, '\\n') == EOF)\n+\t\t\treturn;\n+\t\tif (!*buf->buf)\n+\t\t\tbreak;\n+\t} while (1);\n+\n+\tif (fetch_dumb(nr_heads, to_fetch))\n+\t\texit(128); /* error already reported */\n+\tfree_refs(list_head);\n+\tfree(to_fetch);\n+\n+\tprintf(\"\\n\");\n+\tfflush(stdout);\n+\tstrbuf_reset(buf);\n+}\n+\n int main(int argc, const char **argv)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -110,17 +185,8 @@ int main(int argc, const char **argv)\n \t\tif (strbuf_getline(&buf, stdin, '\\n') == EOF)\n \t\t\tbreak;\n \t\tif (!prefixcmp(buf.buf, \"fetch \")) {\n-\t\t\tchar *obj = buf.buf + strlen(\"fetch \");\n-\t\t\tinit_walker();\n-\t\t\twalker->get_all = 1;\n-\t\t\twalker->get_tree = 1;\n-\t\t\twalker->get_history = 1;\n-\t\t\twalker->get_verbosely = 0;\n-\t\t\twalker->get_recover = 0;\n-\t\t\tif (walker_fetch(walker, 1, &obj, NULL, NULL))\n-\t\t\t\tdie(\"Fetch failed.\");\n-\t\t\tprintf(\"\\n\");\n-\t\t\tfflush(stdout);\n+\t\t\tparse_fetch(&buf);\n+\n \t\t} else if (!strcmp(buf.buf, \"list\")) {\n \t\t\tstruct ref *refs = get_refs();\n \t\t\tstruct ref *posn;\ndiff --git a/transport-helper.c b/transport-helper.c\nindex f57e84c..9de3408 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -10,6 +10,7 @@ struct helper_data\n {\n \tconst char *name;\n \tstruct child_process *helper;\n+\tFILE *out;\n \tunsigned fetch : 1;\n };\n \n@@ -18,7 +19,6 @@ static struct child_process *get_helper(struct transport *transport)\n \tstruct helper_data *data = transport->data;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct child_process *helper;\n-\tFILE *file;\n \n \tif (data->helper)\n \t\treturn data->helper;\n@@ -39,9 +39,9 @@ static struct child_process *get_helper(struct transport *transport)\n \n \twrite_str_in_full(helper->in, \"capabilities\\n\");\n \n-\tfile = xfdopen(helper->out, \"r\");\n+\tdata->out = xfdopen(helper->out, \"r\");\n \twhile (1) {\n-\t\tif (strbuf_getline(&buf, file, '\\n') == EOF)\n+\t\tif (strbuf_getline(&buf, data->out, '\\n') == EOF)\n \t\t\texit(128); /* child died, message supplied already */\n \n \t\tif (!*buf.buf)\n@@ -58,6 +58,7 @@ static int disconnect_helper(struct transport *transport)\n \tif (data->helper) {\n \t\twrite_str_in_full(data->helper->in, \"\\n\");\n \t\tclose(data->helper->in);\n+\t\tfclose(data->out);\n \t\tfinish_command(data->helper);\n \t\tfree((char *)data->helper->argv[0]);\n \t\tfree(data->helper->argv);\n@@ -70,8 +71,7 @@ static int disconnect_helper(struct transport *transport)\n static int fetch_with_fetch(struct transport *transport,\n \t\t\t int nr_heads, const struct ref **to_fetch)\n {\n-\tstruct child_process *helper = get_helper(transport);\n-\tFILE *file = xfdopen(helper->out, \"r\");\n+\tstruct helper_data *data = transport->data;\n \tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n \n@@ -82,12 +82,30 @@ static int fetch_with_fetch(struct transport *transport,\n \n \t\tstrbuf_addf(&buf, \"fetch %s %s\\n\",\n \t\t\t sha1_to_hex(posn->old_sha1), posn->name);\n-\t\twrite_in_full(helper->in, buf.buf, buf.len);\n-\t\tstrbuf_reset(&buf);\n+\t}\n \n-\t\tif (strbuf_getline(&buf, file, '\\n') == EOF)\n+\tstrbuf_addch(&buf, '\\n');\n+\tif (write_in_full(data->helper->in, buf.buf, buf.len) != buf.len)\n+\t\tdie_errno(\"cannot send fetch to %s\", data->name);\n+\n+\twhile (1) {\n+\t\tstrbuf_reset(&buf);\n+\t\tif (strbuf_getline(&buf, data->out, '\\n') == EOF)\n \t\t\texit(128); /* child died, message supplied already */\n+\n+\t\tif (!prefixcmp(buf.buf, \"lock \")) {\n+\t\t\tconst char *name = buf.buf + 5;\n+\t\t\tif (transport->pack_lockfile)\n+\t\t\t\twarning(\"%s also locked %s\", data->name, name);\n+\t\t\telse\n+\t\t\t\ttransport->pack_lockfile = xstrdup(name);\n+\t\t}\n+\t\telse if (!buf.len)\n+\t\t\tbreak;\n+\t\telse\n+\t\t\twarning(\"%s unexpectedly said: '%s'\", data->name, buf.buf);\n \t}\n+\tstrbuf_release(&buf);\n \treturn 0;\n }\n \n@@ -113,21 +131,20 @@ static int fetch(struct transport *transport,\n \n static struct ref *get_refs_list(struct transport *transport, int for_push)\n {\n+\tstruct helper_data *data = transport->data;\n \tstruct child_process *helper;\n \tstruct ref *ret = NULL;\n \tstruct ref **tail = &ret;\n \tstruct ref *posn;\n \tstruct strbuf buf = STRBUF_INIT;\n-\tFILE *file;\n \n \thelper = get_helper(transport);\n \n \twrite_str_in_full(helper->in, \"list\\n\");\n \n-\tfile = xfdopen(helper->out, \"r\");\n \twhile (1) {\n \t\tchar *eov, *eon;\n-\t\tif (strbuf_getline(&buf, file, '\\n') == EOF)\n+\t\tif (strbuf_getline(&buf, data->out, '\\n') == EOF)\n \t\t\texit(128); /* child died, message supplied already */\n \n \t\tif (!*buf.buf)\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126210","messageId":"1256774448-7625-11-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 10/26] remote-helpers: Support custom transport options","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:32Z","receivedAt":"2009-10-29T00:00:32Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Some transports, like the native pack transport implemented by\nfetch-pack, support useful features like depth or include tags.\nThese should be exposed if the underlying helper knows how to\nuse them.\n\nSigned-off-by: Shawn O. Pearce \nCC: Daniel Barkalow \n---\n Documentation/git-remote-helpers.txt | 38 +++++++++++++++\n remote-curl.c | 74 ++++++++++++++++++++++++++++-\n transport-helper.c | 88 +++++++++++++++++++++++++++++++++-\n 3 files changed, 198 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex e44d821..1133f04 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -35,6 +35,16 @@ Commands are given by the caller on the helper's standard input, one per line.\n \tthe name; unrecognized attributes are ignored. After the\n \tcomplete list, outputs a blank line.\n \n+'option' ::\n+\tSet the transport helper option to . Outputs a\n+\tsingle line containing one of 'ok' (option successfully set),\n+\t'unsupported' (option not recognized) or 'error '\n+\t(option is supported but is not correct\n+\tfor it). Options should be set before other commands,\n+\tand may how those commands behave.\n++\n+Supported if the helper has the \"option\" capability.\n+\n 'fetch' ::\n \tFetches the given object, writing the necessary objects\n \tto the database. Fetch commands are sent in a batch, one\n@@ -63,11 +73,39 @@ CAPABILITIES\n 'fetch'::\n \tThis helper supports the 'fetch' command.\n \n+'option'::\n+\tThis helper supports the option command.\n+\n REF LIST ATTRIBUTES\n -------------------\n \n None are defined yet, but the caller must accept any which are supplied.\n \n+OPTIONS\n+-------\n+'option verbosity' ::\n+\tChange the level of messages displayed by the helper.\n+\tWhen N is 0 the end-user has asked the process to be\n+\tquiet, and the helper should produce only error output.\n+\tN of 1 is the default level of verbosity, higher values\n+\tof N correspond to the number of -v flags passed on the\n+\tcommand line.\n+\n+'option progress' \\{'true'|'false'\\}::\n+\tEnable (or disable) progress messages displayed by the\n+\ttransport helper during a command.\n+\n+'option depth' ::\n+\tDeepen the history of a shallow repository.\n+\n+'option followtags' \\{'true'|'false'\\}::\n+\tIf enabled the helper should automatically fetch annotated\n+\ttag objects if the object the tag points at was transferred\n+\tduring the fetch command. If the tag is not fetched by\n+\tthe helper a second fetch command will usually be sent to\n+\task for the tag specifically. Some helpers may be able to\n+\tuse this option to avoid a second network connection.\n+\n Documentation\n -------------\n Documentation by Daniel Barkalow.\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 22cd5c5..0951f11 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -9,12 +9,61 @@ static struct remote *remote;\n static const char *url;\n static struct walker *walker;\n \n+struct options {\n+\tint verbosity;\n+\tunsigned long depth;\n+\tunsigned progress : 1,\n+\t\tfollowtags : 1;\n+};\n+static struct options options;\n+\n static void init_walker(void)\n {\n \tif (!walker)\n \t\twalker = get_http_walker(url, remote);\n }\n \n+static int set_option(const char *name, const char *value)\n+{\n+\tif (!strcmp(name, \"verbosity\")) {\n+\t\tchar *end;\n+\t\tint v = strtol(value, &end, 10);\n+\t\tif (value == end || *end)\n+\t\t\treturn -1;\n+\t\toptions.verbosity = v;\n+\t\treturn 0;\n+\t}\n+\telse if (!strcmp(name, \"progress\")) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.progress = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.progress = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 1 /* TODO implement later */;\n+\t}\n+\telse if (!strcmp(name, \"depth\")) {\n+\t\tchar *end;\n+\t\tunsigned long v = strtoul(value, &end, 10);\n+\t\tif (value == end || *end)\n+\t\t\treturn -1;\n+\t\toptions.depth = v;\n+\t\treturn 1 /* TODO implement later */;\n+\t}\n+\telse if (!strcmp(name, \"followtags\")) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.followtags = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.followtags = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 1 /* TODO implement later */;\n+\t}\n+\telse {\n+\t\treturn 1 /* unsupported */;\n+\t}\n+}\n+\n static struct ref *get_refs(void)\n {\n \tstruct strbuf buffer = STRBUF_INIT;\n@@ -99,7 +148,7 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n \twalker->get_all = 1;\n \twalker->get_tree = 1;\n \twalker->get_history = 1;\n-\twalker->get_verbosely = 0;\n+\twalker->get_verbosely = options.verbosity >= 3;\n \twalker->get_recover = 0;\n \tret = walker_fetch(walker, nr_heads, targets, NULL, NULL);\n \n@@ -173,6 +222,9 @@ int main(int argc, const char **argv)\n \t\treturn 1;\n \t}\n \n+\toptions.verbosity = 1;\n+\toptions.progress = !!isatty(2);\n+\n \tremote = remote_get(argv[1]);\n \n \tif (argc > 2) {\n@@ -198,8 +250,28 @@ int main(int argc, const char **argv)\n \t\t\t}\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n+\t\t} else if (!prefixcmp(buf.buf, \"option \")) {\n+\t\t\tchar *name = buf.buf + strlen(\"option \");\n+\t\t\tchar *value = strchr(name, ' ');\n+\t\t\tint result;\n+\n+\t\t\tif (value)\n+\t\t\t\t*value++ = '\\0';\n+\t\t\telse\n+\t\t\t\tvalue = \"true\";\n+\n+\t\t\tresult = set_option(name, value);\n+\t\t\tif (!result)\n+\t\t\t\tprintf(\"ok\\n\");\n+\t\t\telse if (result < 0)\n+\t\t\t\tprintf(\"error invalid value\\n\");\n+\t\t\telse\n+\t\t\t\tprintf(\"unsupported\\n\");\n+\t\t\tfflush(stdout);\n+\n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n \t\t\tprintf(\"fetch\\n\");\n+\t\t\tprintf(\"option\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n \t\t} else {\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 9de3408..577abc6 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -5,13 +5,15 @@\n #include \"commit.h\"\n #include \"diff.h\"\n #include \"revision.h\"\n+#include \"quote.h\"\n \n struct helper_data\n {\n \tconst char *name;\n \tstruct child_process *helper;\n \tFILE *out;\n-\tunsigned fetch : 1;\n+\tunsigned fetch : 1,\n+\t\toption : 1;\n };\n \n static struct child_process *get_helper(struct transport *transport)\n@@ -48,6 +50,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\tbreak;\n \t\tif (!strcmp(buf.buf, \"fetch\"))\n \t\t\tdata->fetch = 1;\n+\t\tif (!strcmp(buf.buf, \"option\"))\n+\t\t\tdata->option = 1;\n \t}\n \treturn data->helper;\n }\n@@ -65,9 +69,88 @@ static int disconnect_helper(struct transport *transport)\n \t\tfree(data->helper);\n \t\tdata->helper = NULL;\n \t}\n+\tfree(data);\n \treturn 0;\n }\n \n+static const char *unsupported_options[] = {\n+\tTRANS_OPT_UPLOADPACK,\n+\tTRANS_OPT_RECEIVEPACK,\n+\tTRANS_OPT_THIN,\n+\tTRANS_OPT_KEEP\n+\t};\n+static const char *boolean_options[] = {\n+\tTRANS_OPT_THIN,\n+\tTRANS_OPT_KEEP,\n+\tTRANS_OPT_FOLLOWTAGS\n+\t};\n+\n+static int set_helper_option(struct transport *transport,\n+\t\t\t const char *name, const char *value)\n+{\n+\tstruct helper_data *data = transport->data;\n+\tstruct child_process *helper = get_helper(transport);\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint i, ret, is_bool = 0;\n+\n+\tif (!data->option)\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(unsupported_options); i++) {\n+\t\tif (!strcmp(name, unsupported_options[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\tfor (i = 0; i < ARRAY_SIZE(boolean_options); i++) {\n+\t\tif (!strcmp(name, boolean_options[i])) {\n+\t\t\tis_bool = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tstrbuf_addf(&buf, \"option %s \", name);\n+\tif (is_bool)\n+\t\tstrbuf_addstr(&buf, value ? \"true\" : \"false\");\n+\telse\n+\t\tquote_c_style(value, &buf, NULL, 0);\n+\tstrbuf_addch(&buf, '\\n');\n+\n+\tif (write_in_full(helper->in, buf.buf, buf.len) != buf.len)\n+\t\tdie_errno(\"cannot send option to %s\", data->name);\n+\n+\tstrbuf_reset(&buf);\n+\tif (strbuf_getline(&buf, data->out, '\\n') == EOF)\n+\t\texit(128); /* child died, message supplied already */\n+\n+\tif (!strcmp(buf.buf, \"ok\"))\n+\t\tret = 0;\n+\telse if (!prefixcmp(buf.buf, \"error\")) {\n+\t\tret = -1;\n+\t} else if (!strcmp(buf.buf, \"unsupported\"))\n+\t\tret = 1;\n+\telse {\n+\t\twarning(\"%s unexpectedly said: '%s'\", data->name, buf.buf);\n+\t\tret = 1;\n+\t}\n+\tstrbuf_release(&buf);\n+\treturn ret;\n+}\n+\n+static void standard_options(struct transport *t)\n+{\n+\tchar buf[16];\n+\tint n;\n+\tint v = t->verbose;\n+\tint no_progress = v < 0 || (!t->progress && !isatty(1));\n+\n+\tset_helper_option(t, \"progress\", !no_progress ? \"true\" : \"false\");\n+\n+\tn = snprintf(buf, sizeof(buf), \"%d\", v + 1);\n+\tif (n >= sizeof(buf))\n+\t\tdie(\"impossibly large verbosity value\");\n+\tset_helper_option(t, \"verbosity\", buf);\n+}\n+\n static int fetch_with_fetch(struct transport *transport,\n \t\t\t int nr_heads, const struct ref **to_fetch)\n {\n@@ -75,6 +158,8 @@ static int fetch_with_fetch(struct transport *transport,\n \tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n \n+\tstandard_options(transport);\n+\n \tfor (i = 0; i < nr_heads; i++) {\n \t\tconst struct ref *posn = to_fetch[i];\n \t\tif (posn->status & REF_STATUS_UPTODATE)\n@@ -178,6 +263,7 @@ int transport_helper_init(struct transport *transport, const char *name)\n \tdata->name = name;\n \n \ttransport->data = data;\n+\ttransport->set_option = set_helper_option;\n \ttransport->get_refs_list = get_refs_list;\n \ttransport->fetch = fetch;\n \ttransport->disconnect = disconnect_helper;\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126199","messageId":"1256774448-7625-12-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 11/26] Move WebDAV HTTP push under remote-curl","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:33Z","receivedAt":"2009-10-29T00:00:33Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The remote helper interface now supports the push capability,\nwhich can be used to ask the implementation to push one or more\nspecs to the remote repository. For remote-curl we implement this\nby calling the existing WebDAV based git-http-push executable.\n\nInternally the helper interface uses the push_refs transport hook\nso that the complexity of the refspec parsing and matching can be\nreused between remote implementations. When possible however the\nhelper protocol uses source ref name rather than the source SHA-1,\nthereby allowing the helper to access this name if it is useful.\n\n>From Clemens Buchacher :\n update http tests according to remote-curl capabilities\n\n o Pushing packed refs is now fixed.\n\n o The transport helper fails if refs are already up-to-date. Add\n a test for that.\n\n o The transport helper will notice if refs are already\n up-to-date. We therefore need to update server info in the\n unpacked-refs test.\n\n o The transport helper will purge deleted branches automatically.\n\nSigned-off-by: Tay Ray Chuan \nSigned-off-by: Clemens Buchacher \nSigned-off-by: Shawn O. Pearce \nCC: Daniel Barkalow \nCC: Mike Hommey \n---\n Documentation/git-remote-helpers.txt | 33 ++++++++-\n http-push.c | 29 +++++++-\n remote-curl.c | 97 +++++++++++++++++++++---\n t/t5540-http-push.sh | 14 ++-\n transport-helper.c | 137 +++++++++++++++++++++++++++++++++-\n transport.c | 31 --------\n 6 files changed, 287 insertions(+), 54 deletions(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex 1133f04..8beb42d 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -34,6 +34,10 @@ Commands are given by the caller on the helper's standard input, one per line.\n \tvalue of the ref. A space-separated list of attributes follows\n \tthe name; unrecognized attributes are ignored. After the\n \tcomplete list, outputs a blank line.\n++\n+If 'push' is supported this may be called as 'list for-push'\n+to obtain the current refs prior to sending one or more 'push'\n+commands to the helper.\n \n 'option' ::\n \tSet the transport helper option to . Outputs a\n@@ -59,6 +63,22 @@ suitably updated.\n +\n Supported if the helper has the \"fetch\" capability.\n \n+'push' +:::\n+\tPushes the given commit or branch locally to the\n+\tremote branch described by . A batch sequence of\n+\tone or more push commands is terminated with a blank line.\n++\n+Zero or more protocol options may be entered after the last 'push'\n+command, before the batch's terminating blank line.\n++\n+When the push is complete, outputs one or more 'ok ' or\n+'error ?' lines to indicate success or failure of\n+each pushed ref. The status report output is terminated by\n+a blank line. The option field may be quoted in a C\n+style string if it contains an LF.\n++\n+Supported if the helper has the \"push\" capability.\n+\n If a fatal error occurs, the program writes the error message to\n stderr and exits. The caller should expect that a suitable error\n message has been printed if the child closes the connection without\n@@ -76,10 +96,16 @@ CAPABILITIES\n 'option'::\n \tThis helper supports the option command.\n \n+'push'::\n+\tThis helper supports the 'push' command.\n+\n REF LIST ATTRIBUTES\n -------------------\n \n-None are defined yet, but the caller must accept any which are supplied.\n+'for-push'::\n+\tThe caller wants to use the ref list to prepare push\n+\tcommands. A helper might chose to acquire the ref list by\n+\topening a different type of connection to the destination.\n \n OPTIONS\n -------\n@@ -106,6 +132,11 @@ OPTIONS\n \task for the tag specifically. Some helpers may be able to\n \tuse this option to avoid a second network connection.\n \n+'option dry-run' \\{'true'|'false'\\}:\n+\tIf true, pretend the operation completed successfully,\n+\tbut don't actually change any repository data.\tFor most\n+\thelpers this only applies to the 'push', if supported.\n+\n Documentation\n -------------\n Documentation by Daniel Barkalow.\ndiff --git a/http-push.c b/http-push.c\nindex cc5d4b8..f10803a 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -78,6 +78,7 @@ static int push_verbosely;\n static int push_all = MATCH_REFS_NONE;\n static int force_all;\n static int dry_run;\n+static int helper_status;\n \n static struct object_list *objects;\n \n@@ -1813,6 +1814,10 @@ int main(int argc, char **argv)\n \t\t\t\tdry_run = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(arg, \"--helper-status\")) {\n+\t\t\t\thelper_status = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(arg, \"--verbose\")) {\n \t\t\t\tpush_verbosely = 1;\n \t\t\t\thttp_is_verbose = 1;\n@@ -1911,9 +1916,12 @@ int main(int argc, char **argv)\n \n \t/* Remove a remote branch if -d or -D was specified */\n \tif (delete_branch) {\n-\t\tif (delete_remote_branch(refspec[0], force_delete) == -1)\n+\t\tif (delete_remote_branch(refspec[0], force_delete) == -1) {\n \t\t\tfprintf(stderr, \"Unable to delete remote branch %s\\n\",\n \t\t\t\trefspec[0]);\n+\t\t\tif (helper_status)\n+\t\t\t\tprintf(\"error %s cannot remove\\n\", refspec[0]);\n+\t\t}\n \t\tgoto cleanup;\n \t}\n \n@@ -1925,6 +1933,8 @@ int main(int argc, char **argv)\n \t}\n \tif (!remote_refs) {\n \t\tfprintf(stderr, \"No refs in common and none specified; doing nothing.\\n\");\n+\t\tif (helper_status)\n+\t\t\tprintf(\"error null no match\\n\");\n \t\trc = 0;\n \t\tgoto cleanup;\n \t}\n@@ -1942,8 +1952,12 @@ int main(int argc, char **argv)\n \t\tif (is_null_sha1(ref->peer_ref->new_sha1)) {\n \t\t\tif (delete_remote_branch(ref->name, 1) == -1) {\n \t\t\t\terror(\"Could not remove %s\", ref->name);\n+\t\t\t\tif (helper_status)\n+\t\t\t\t\tprintf(\"error %s cannot remove\\n\", ref->name);\n \t\t\t\trc = -4;\n \t\t\t}\n+\t\t\telse if (helper_status)\n+\t\t\t\tprintf(\"ok %s\\n\", ref->name);\n \t\t\tnew_refs++;\n \t\t\tcontinue;\n \t\t}\n@@ -1951,6 +1965,8 @@ int main(int argc, char **argv)\n \t\tif (!hashcmp(ref->old_sha1, ref->peer_ref->new_sha1)) {\n \t\t\tif (push_verbosely || 1)\n \t\t\t\tfprintf(stderr, \"'%s': up-to-date\\n\", ref->name);\n+\t\t\tif (helper_status)\n+\t\t\t\tprintf(\"ok %s up to date\\n\", ref->name);\n \t\t\tcontinue;\n \t\t}\n \n@@ -1974,6 +1990,8 @@ int main(int argc, char **argv)\n \t\t\t\t \"need to pull first?\",\n \t\t\t\t ref->name,\n \t\t\t\t ref->peer_ref->name);\n+\t\t\t\tif (helper_status)\n+\t\t\t\t\tprintf(\"error %s non-fast forward\\n\", ref->name);\n \t\t\t\trc = -2;\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -1987,14 +2005,19 @@ int main(int argc, char **argv)\n \t\tif (strcmp(ref->name, ref->peer_ref->name))\n \t\t\tfprintf(stderr, \" using '%s'\", ref->peer_ref->name);\n \t\tfprintf(stderr, \"\\n from %s\\n to %s\\n\", old_hex, new_hex);\n-\t\tif (dry_run)\n+\t\tif (dry_run) {\n+\t\t\tif (helper_status)\n+\t\t\t\tprintf(\"ok %s\\n\", ref->name);\n \t\t\tcontinue;\n+\t\t}\n \n \t\t/* Lock remote branch ref */\n \t\tref_lock = lock_remote(ref->name, LOCK_TIME);\n \t\tif (ref_lock == NULL) {\n \t\t\tfprintf(stderr, \"Unable to lock remote branch %s\\n\",\n \t\t\t\tref->name);\n+\t\t\tif (helper_status)\n+\t\t\t\tprintf(\"error %s lock error\\n\", ref->name);\n \t\t\trc = 1;\n \t\t\tcontinue;\n \t\t}\n@@ -2045,6 +2068,8 @@ int main(int argc, char **argv)\n \n \t\tif (!rc)\n \t\t\tfprintf(stderr, \" done\\n\");\n+\t\tif (helper_status)\n+\t\t\tprintf(\"%s %s\\n\", !rc ? \"ok\" : \"error\", ref->name);\n \t\tunlock_remote(ref_lock);\n \t\tcheck_locks();\n \t}\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 0951f11..5c9dd97 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -4,6 +4,7 @@\n #include \"walker.h\"\n #include \"http.h\"\n #include \"exec_cmd.h\"\n+#include \"run-command.h\"\n \n static struct remote *remote;\n static const char *url;\n@@ -13,7 +14,8 @@ struct options {\n \tint verbosity;\n \tunsigned long depth;\n \tunsigned progress : 1,\n-\t\tfollowtags : 1;\n+\t\tfollowtags : 1,\n+\t\tdry_run : 1;\n };\n static struct options options;\n \n@@ -59,6 +61,15 @@ static int set_option(const char *name, const char *value)\n \t\t\treturn -1;\n \t\treturn 1 /* TODO implement later */;\n \t}\n+\telse if (!strcmp(name, \"dry-run\")) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.dry_run = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.dry_run = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n+\t}\n \telse {\n \t\treturn 1 /* unsupported */;\n \t}\n@@ -136,6 +147,20 @@ static struct ref *get_refs(void)\n \treturn refs;\n }\n \n+static void output_refs(struct ref *refs)\n+{\n+\tstruct ref *posn;\n+\tfor (posn = refs; posn; posn = posn->next) {\n+\t\tif (posn->symref)\n+\t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n+\t\telse\n+\t\t\tprintf(\"%s %s\\n\", sha1_to_hex(posn->old_sha1), posn->name);\n+\t}\n+\tprintf(\"\\n\");\n+\tfflush(stdout);\n+\tfree_refs(refs);\n+}\n+\n static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n {\n \tchar **targets = xmalloc(nr_heads * sizeof(char*));\n@@ -211,6 +236,58 @@ static void parse_fetch(struct strbuf *buf)\n \tstrbuf_reset(buf);\n }\n \n+static int push_dav(int nr_spec, char **specs)\n+{\n+\tconst char **argv = xmalloc((10 + nr_spec) * sizeof(char*));\n+\tint argc = 0, i;\n+\n+\targv[argc++] = \"http-push\";\n+\targv[argc++] = \"--helper-status\";\n+\tif (options.dry_run)\n+\t\targv[argc++] = \"--dry-run\";\n+\tif (options.verbosity > 1)\n+\t\targv[argc++] = \"--verbose\";\n+\targv[argc++] = url;\n+\tfor (i = 0; i < nr_spec; i++)\n+\t\targv[argc++] = specs[i];\n+\targv[argc++] = NULL;\n+\n+\tif (run_command_v_opt(argv, RUN_GIT_CMD))\n+\t\tdie(\"git-%s failed\", argv[0]);\n+\tfree(argv);\n+\treturn 0;\n+}\n+\n+static void parse_push(struct strbuf *buf)\n+{\n+\tchar **specs = NULL;\n+\tint alloc_spec = 0, nr_spec = 0, i;\n+\n+\tdo {\n+\t\tif (!prefixcmp(buf->buf, \"push \")) {\n+\t\t\tALLOC_GROW(specs, nr_spec + 1, alloc_spec);\n+\t\t\tspecs[nr_spec++] = xstrdup(buf->buf + 5);\n+\t\t}\n+\t\telse\n+\t\t\tdie(\"http transport does not support %s\", buf->buf);\n+\n+\t\tstrbuf_reset(buf);\n+\t\tif (strbuf_getline(buf, stdin, '\\n') == EOF)\n+\t\t\treturn;\n+\t\tif (!*buf->buf)\n+\t\t\tbreak;\n+\t} while (1);\n+\n+\tif (push_dav(nr_spec, specs))\n+\t\texit(128); /* error already reported */\n+\tfor (i = 0; i < nr_spec; i++)\n+\t\tfree(specs[i]);\n+\tfree(specs);\n+\n+\tprintf(\"\\n\");\n+\tfflush(stdout);\n+}\n+\n int main(int argc, const char **argv)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -239,17 +316,12 @@ int main(int argc, const char **argv)\n \t\tif (!prefixcmp(buf.buf, \"fetch \")) {\n \t\t\tparse_fetch(&buf);\n \n-\t\t} else if (!strcmp(buf.buf, \"list\")) {\n-\t\t\tstruct ref *refs = get_refs();\n-\t\t\tstruct ref *posn;\n-\t\t\tfor (posn = refs; posn; posn = posn->next) {\n-\t\t\t\tif (posn->symref)\n-\t\t\t\t\tprintf(\"@%s %s\\n\", posn->symref, posn->name);\n-\t\t\t\telse\n-\t\t\t\t\tprintf(\"%s %s\\n\", sha1_to_hex(posn->old_sha1), posn->name);\n-\t\t\t}\n-\t\t\tprintf(\"\\n\");\n-\t\t\tfflush(stdout);\n+\t\t} else if (!strcmp(buf.buf, \"list\") || !prefixcmp(buf.buf, \"list \")) {\n+\t\t\toutput_refs(get_refs());\n+\n+\t\t} else if (!prefixcmp(buf.buf, \"push \")) {\n+\t\t\tparse_push(&buf);\n+\n \t\t} else if (!prefixcmp(buf.buf, \"option \")) {\n \t\t\tchar *name = buf.buf + strlen(\"option \");\n \t\t\tchar *value = strchr(name, ' ');\n@@ -272,6 +344,7 @@ int main(int argc, const char **argv)\n \t\t} else if (!strcmp(buf.buf, \"capabilities\")) {\n \t\t\tprintf(\"fetch\\n\");\n \t\t\tprintf(\"option\\n\");\n+\t\t\tprintf(\"push\\n\");\n \t\t\tprintf(\"\\n\");\n \t\t\tfflush(stdout);\n \t\t} else {\ndiff --git a/t/t5540-http-push.sh b/t/t5540-http-push.sh\nindex f4a2cf6..09edd23 100755\n--- a/t/t5540-http-push.sh\n+++ b/t/t5540-http-push.sh\n@@ -36,6 +36,7 @@ test_expect_success 'setup remote repository' '\n \tcd test_repo.git &&\n \tgit --bare update-server-info &&\n \tmv hooks/post-update.sample hooks/post-update &&\n+\tORIG_HEAD=$(git rev-parse --verify HEAD) &&\n \tcd - &&\n \tmv test_repo.git \"$HTTPD_DOCUMENT_ROOT_PATH\"\n '\n@@ -45,7 +46,7 @@ test_expect_success 'clone remote repository' '\n \tgit clone $HTTPD_URL/test_repo.git test_repo_clone\n '\n \n-test_expect_failure 'push to remote repository with packed refs' '\n+test_expect_success 'push to remote repository with packed refs' '\n \tcd \"$ROOT_PATH\"/test_repo_clone &&\n \t: >path2 &&\n \tgit add path2 &&\n@@ -57,11 +58,15 @@ test_expect_failure 'push to remote repository with packed refs' '\n \t test $HEAD = $(git rev-parse --verify HEAD))\n '\n \n-test_expect_success ' push to remote repository with unpacked refs' '\n+test_expect_failure 'push already up-to-date' '\n+\tgit push\n+'\n+\n+test_expect_success 'push to remote repository with unpacked refs' '\n \t(cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n \t rm packed-refs &&\n-\t git update-ref refs/heads/master \\\n-\t\t0c973ae9bd51902a28466f3850b543fa66a6aaf4) &&\n+\t git update-ref refs/heads/master $ORIG_HEAD &&\n+\t git --bare update-server-info) &&\n \tgit push &&\n \t(cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n \t test $HEAD = $(git rev-parse --verify HEAD))\n@@ -113,7 +118,6 @@ test_expect_success 'create and delete remote branch' '\n \tgit push origin dev &&\n \tgit fetch &&\n \tgit push origin :dev &&\n-\tgit branch -d -r origin/dev &&\n \tgit fetch &&\n \ttest_must_fail git show-ref --verify refs/remotes/origin/dev\n '\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 577abc6..16c6641 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1,6 +1,6 @@\n #include \"cache.h\"\n #include \"transport.h\"\n-\n+#include \"quote.h\"\n #include \"run-command.h\"\n #include \"commit.h\"\n #include \"diff.h\"\n@@ -13,7 +13,8 @@ struct helper_data\n \tstruct child_process *helper;\n \tFILE *out;\n \tunsigned fetch : 1,\n-\t\toption : 1;\n+\t\toption : 1,\n+\t\tpush : 1;\n };\n \n static struct child_process *get_helper(struct transport *transport)\n@@ -52,6 +53,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\tdata->fetch = 1;\n \t\tif (!strcmp(buf.buf, \"option\"))\n \t\t\tdata->option = 1;\n+\t\tif (!strcmp(buf.buf, \"push\"))\n+\t\t\tdata->push = 1;\n \t}\n \treturn data->helper;\n }\n@@ -214,6 +217,130 @@ static int fetch(struct transport *transport,\n \treturn -1;\n }\n \n+static int push_refs(struct transport *transport,\n+\t\tstruct ref *remote_refs, int flags)\n+{\n+\tint force_all = flags & TRANSPORT_PUSH_FORCE;\n+\tint mirror = flags & TRANSPORT_PUSH_MIRROR;\n+\tstruct helper_data *data = transport->data;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct child_process *helper;\n+\tstruct ref *ref;\n+\n+\tif (!remote_refs)\n+\t\treturn 0;\n+\n+\thelper = get_helper(transport);\n+\tif (!data->push)\n+\t\treturn 1;\n+\n+\tfor (ref = remote_refs; ref; ref = ref->next) {\n+\t\tif (ref->peer_ref)\n+\t\t\thashcpy(ref->new_sha1, ref->peer_ref->new_sha1);\n+\t\telse if (!mirror)\n+\t\t\tcontinue;\n+\n+\t\tref->deletion = is_null_sha1(ref->new_sha1);\n+\t\tif (!ref->deletion &&\n+\t\t\t!hashcmp(ref->old_sha1, ref->new_sha1)) {\n+\t\t\tref->status = REF_STATUS_UPTODATE;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (force_all)\n+\t\t\tref->force = 1;\n+\n+\t\tstrbuf_addstr(&buf, \"push \");\n+\t\tif (!ref->deletion) {\n+\t\t\tif (ref->force)\n+\t\t\t\tstrbuf_addch(&buf, '+');\n+\t\t\tif (ref->peer_ref)\n+\t\t\t\tstrbuf_addstr(&buf, ref->peer_ref->name);\n+\t\t\telse\n+\t\t\t\tstrbuf_addstr(&buf, sha1_to_hex(ref->new_sha1));\n+\t\t}\n+\t\tstrbuf_addch(&buf, ':');\n+\t\tstrbuf_addstr(&buf, ref->name);\n+\t\tstrbuf_addch(&buf, '\\n');\n+\t}\n+\n+\ttransport->verbose = flags & TRANSPORT_PUSH_VERBOSE ? 1 : 0;\n+\tstandard_options(transport);\n+\n+\tif (flags & TRANSPORT_PUSH_DRY_RUN) {\n+\t\tif (set_helper_option(transport, \"dry-run\", \"true\") != 0)\n+\t\t\tdie(\"helper %s does not support dry-run\", data->name);\n+\t}\n+\n+\tstrbuf_addch(&buf, '\\n');\n+\tif (write_in_full(helper->in, buf.buf, buf.len) != buf.len)\n+\t\texit(128);\n+\n+\tref = remote_refs;\n+\twhile (1) {\n+\t\tchar *refname, *msg;\n+\t\tint status;\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tif (strbuf_getline(&buf, data->out, '\\n') == EOF)\n+\t\t\texit(128); /* child died, message supplied already */\n+\t\tif (!buf.len)\n+\t\t\tbreak;\n+\n+\t\tif (!prefixcmp(buf.buf, \"ok \")) {\n+\t\t\tstatus = REF_STATUS_OK;\n+\t\t\trefname = buf.buf + 3;\n+\t\t} else if (!prefixcmp(buf.buf, \"error \")) {\n+\t\t\tstatus = REF_STATUS_REMOTE_REJECT;\n+\t\t\trefname = buf.buf + 6;\n+\t\t} else\n+\t\t\tdie(\"expected ok/error, helper said '%s'\\n\", buf.buf);\n+\n+\t\tmsg = strchr(refname, ' ');\n+\t\tif (msg) {\n+\t\t\tstruct strbuf msg_buf = STRBUF_INIT;\n+\t\t\tconst char *end;\n+\n+\t\t\t*msg++ = '\\0';\n+\t\t\tif (!unquote_c_style(&msg_buf, msg, &end))\n+\t\t\t\tmsg = strbuf_detach(&msg_buf, NULL);\n+\t\t\telse\n+\t\t\t\tmsg = xstrdup(msg);\n+\t\t\tstrbuf_release(&msg_buf);\n+\n+\t\t\tif (!strcmp(msg, \"no match\")) {\n+\t\t\t\tstatus = REF_STATUS_NONE;\n+\t\t\t\tfree(msg);\n+\t\t\t\tmsg = NULL;\n+\t\t\t}\n+\t\t\telse if (!strcmp(msg, \"up to date\")) {\n+\t\t\t\tstatus = REF_STATUS_UPTODATE;\n+\t\t\t\tfree(msg);\n+\t\t\t\tmsg = NULL;\n+\t\t\t}\n+\t\t\telse if (!strcmp(msg, \"non-fast forward\")) {\n+\t\t\t\tstatus = REF_STATUS_REJECT_NONFASTFORWARD;\n+\t\t\t\tfree(msg);\n+\t\t\t\tmsg = NULL;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (ref)\n+\t\t\tref = find_ref_by_name(ref, refname);\n+\t\tif (!ref)\n+\t\t\tref = find_ref_by_name(remote_refs, refname);\n+\t\tif (!ref) {\n+\t\t\twarning(\"helper reported unexpected status of %s\", refname);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tref->status = status;\n+\t\tref->remote_status = msg;\n+\t}\n+\tstrbuf_release(&buf);\n+\treturn 0;\n+}\n+\n static struct ref *get_refs_list(struct transport *transport, int for_push)\n {\n \tstruct helper_data *data = transport->data;\n@@ -225,7 +352,10 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)\n \n \thelper = get_helper(transport);\n \n-\twrite_str_in_full(helper->in, \"list\\n\");\n+\tif (data->push && for_push)\n+\t\twrite_str_in_full(helper->in, \"list for-push\\n\");\n+\telse\n+\t\twrite_str_in_full(helper->in, \"list\\n\");\n \n \twhile (1) {\n \t\tchar *eov, *eon;\n@@ -266,6 +396,7 @@ int transport_helper_init(struct transport *transport, const char *name)\n \ttransport->set_option = set_helper_option;\n \ttransport->get_refs_list = get_refs_list;\n \ttransport->fetch = fetch;\n+\ttransport->push_refs = push_refs;\n \ttransport->disconnect = disconnect_helper;\n \treturn 0;\n }\ndiff --git a/transport.c b/transport.c\nindex 644a30a..6d9652d 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -349,35 +349,6 @@ static int rsync_transport_push(struct transport *transport,\n \treturn result;\n }\n \n-#ifndef NO_CURL\n-static int curl_transport_push(struct transport *transport, int refspec_nr, const char **refspec, int flags)\n-{\n-\tconst char **argv;\n-\tint argc;\n-\n-\tif (flags & TRANSPORT_PUSH_MIRROR)\n-\t\treturn error(\"http transport does not support mirror mode\");\n-\n-\targv = xmalloc((refspec_nr + 12) * sizeof(char *));\n-\targv[0] = \"http-push\";\n-\targc = 1;\n-\tif (flags & TRANSPORT_PUSH_ALL)\n-\t\targv[argc++] = \"--all\";\n-\tif (flags & TRANSPORT_PUSH_FORCE)\n-\t\targv[argc++] = \"--force\";\n-\tif (flags & TRANSPORT_PUSH_DRY_RUN)\n-\t\targv[argc++] = \"--dry-run\";\n-\tif (flags & TRANSPORT_PUSH_VERBOSE)\n-\t\targv[argc++] = \"--verbose\";\n-\targv[argc++] = transport->url;\n-\twhile (refspec_nr--)\n-\t\targv[argc++] = *refspec++;\n-\targv[argc] = NULL;\n-\treturn !!run_command_v_opt(argv, RUN_GIT_CMD);\n-}\n-\n-#endif\n-\n struct bundle_transport_data {\n \tint fd;\n \tstruct bundle_header header;\n@@ -826,8 +797,6 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \t\ttransport_helper_init(ret, \"curl\");\n #ifdef NO_CURL\n \t\terror(\"git was compiled without libcurl support.\");\n-#else\n-\t\tret->push = curl_transport_push;\n #endif\n \n \t} else if (is_local(url) && is_file(url)) {\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126197","messageId":"1256774448-7625-13-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 12/26] remote-helpers: return successfully if everything up-to-date","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:34Z","receivedAt":"2009-10-29T00:00:34Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"From: Clemens Buchacher \n\nSigned-off-by: Clemens Buchacher \nSigned-off-by: Shawn O. Pearce \n---\n t/t5540-http-push.sh | 2 +-\n transport-helper.c | 2 ++\n 2 files changed, 3 insertions(+), 1 deletions(-)\n\ndiff --git a/t/t5540-http-push.sh b/t/t5540-http-push.sh\nindex 09edd23..2ece661 100755\n--- a/t/t5540-http-push.sh\n+++ b/t/t5540-http-push.sh\n@@ -58,7 +58,7 @@ test_expect_success 'push to remote repository with packed refs' '\n \t test $HEAD = $(git rev-parse --verify HEAD))\n '\n \n-test_expect_failure 'push already up-to-date' '\n+test_expect_success 'push already up-to-date' '\n \tgit push\n '\n \ndiff --git a/transport-helper.c b/transport-helper.c\nindex 16c6641..5078c71 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -263,6 +263,8 @@ static int push_refs(struct transport *transport,\n \t\tstrbuf_addstr(&buf, ref->name);\n \t\tstrbuf_addch(&buf, '\\n');\n \t}\n+\tif (buf.len == 0)\n+\t\treturn 0;\n \n \ttransport->verbose = flags & TRANSPORT_PUSH_VERBOSE ? 1 : 0;\n \tstandard_options(transport);\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126213","messageId":"1256774448-7625-14-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 13/26] Git-aware CGI to provide dumb HTTP transport","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:35Z","receivedAt":"2009-10-29T00:00:35Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The git-http-backend CGI can be configured into any Apache server\nusing ScriptAlias, such as with the following configuration:\n\n LoadModule cgi_module /usr/libexec/apache2/mod_cgi.so\n LoadModule alias_module /usr/libexec/apache2/mod_alias.so\n ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/\n\nRepositories are accessed via the translated PATH_INFO.\n\nThe CGI is backwards compatible with the dumb client, allowing all\nolder HTTP clients to continue to download repositories which are\nmanaged by the CGI.\n\nSigned-off-by: Shawn O. Pearce \n---\n .gitignore | 1 +\n Documentation/git-http-backend.txt | 105 +++++++++++++\n Makefile | 1 +\n http-backend.c | 290 ++++++++++++++++++++++++++++++++++++\n 4 files changed, 397 insertions(+), 0 deletions(-)\n create mode 100644 Documentation/git-http-backend.txt\n create mode 100644 http-backend.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 51a37b1..353d22f 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -55,6 +55,7 @@ git-get-tar-commit-id\n git-grep\n git-hash-object\n git-help\n+git-http-backend\n git-http-fetch\n git-http-push\n git-imap-send\ndiff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt\nnew file mode 100644\nindex 0000000..867675f\n--- /dev/null\n+++ b/Documentation/git-http-backend.txt\n@@ -0,0 +1,105 @@\n+git-http-backend(1)\n+===================\n+\n+NAME\n+----\n+git-http-backend - Server side implementation of Git over HTTP\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git-http-backend'\n+\n+DESCRIPTION\n+-----------\n+A simple CGI program to serve the contents of a Git repository to Git\n+clients accessing the repository over http:// and https:// protocols.\n+\n+By default, only the `upload-pack` service is enabled, which serves\n+'git-fetch-pack' and 'git-ls-remote' clients, which are invoked from\n+'git-fetch', 'git-pull', and 'git-clone'.\n+\n+This is ideally suited for read-only updates, i.e., pulling from\n+git repositories.\n+\n+URL TRANSLATION\n+---------------\n+'git-http-backend' relies on the invoking web server to perform\n+URL to path translation, and store the repository path into the\n+PATH_TRANSLATED environment variable. Most web servers will do\n+this translation automatically, resolving the suffix after the\n+CGI name relative to the server's document root.\n+\n+EXAMPLES\n+--------\n+\n+Apache 2.x::\n+\tTo serve all Git repositories contained within the '/git/'\n+\tsubdirectory of the DocumentRoot, ensure mod_cgi and\n+\tmod_alias are enabled, and create a ScriptAlias to the CGI:\n++\n+----------------------------------------------------------------\n+ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/git/\n+\n+\n+\tOptions None\n+\n+\n+\tOptions ExecCGI\n+\n+----------------------------------------------------------------\n++\n+To require authentication for reads, use a Directory\n+directive around the repository, or one of its parent directories:\n++\n+----------------------------------------------------------------\n+\n+\tAuthType Basic\n+\tAuthName \"Private Git Access\"\n+\tRequire group committers\n+\t...\n+\n+----------------------------------------------------------------\n+\n+Accelerated static Apache 2.x::\n+\tSimilar to the above, but Apache can be used to return static\n+\tfiles that are stored on disk.\tOn many systems this may\n+\tbe more efficient as Apache can ask the kernel to copy the\n+\tfile contents from the file system directly to the network:\n++\n+----------------------------------------------------------------\n+DocumentRoot /var/www\n+\n+ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/git/\n+Alias /git_static/ /var/www/git/\n+\n+RewriteEngine on\n+RewriteRule ^/git/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /git_static/$1 [PT]\n+RewriteRule ^/git/(.*/objects/pack/pack-[0-9a-f]{40}.pack)$ /git_static/$1 [PT]\n+RewriteRule ^/git/(.*/objects/pack/pack-[0-9a-f]{40}.idx)$ /git_static/$1 [PT]\n+----------------------------------------------------------------\n+\n+\n+ENVIRONMENT\n+-----------\n+'git-http-backend' relies upon the CGI environment variables set\n+by the invoking web server, including:\n+\n+* PATH_TRANSLATED\n+* REMOTE_USER\n+* REMOTE_ADDR\n+* CONTENT_TYPE\n+* QUERY_STRING\n+* REQUEST_METHOD\n+\n+Author\n+------\n+Written by Shawn O. Pearce .\n+\n+Documentation\n+--------------\n+Documentation by Shawn O. Pearce .\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex fea237b..271c290 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -365,6 +365,7 @@ PROGRAMS += git-show-index$X\n PROGRAMS += git-unpack-file$X\n PROGRAMS += git-upload-pack$X\n PROGRAMS += git-var$X\n+PROGRAMS += git-http-backend$X\n \n # List built-in command $C whose implementation cmd_$C() is not in\n # builtin-$C.o but is linked in as part of some other command.\ndiff --git a/http-backend.c b/http-backend.c\nnew file mode 100644\nindex 0000000..374f60d\n--- /dev/null\n+++ b/http-backend.c\n@@ -0,0 +1,290 @@\n+#include \"cache.h\"\n+#include \"refs.h\"\n+#include \"pkt-line.h\"\n+#include \"object.h\"\n+#include \"tag.h\"\n+#include \"exec_cmd.h\"\n+\n+static const char content_type[] = \"Content-Type\";\n+static const char content_length[] = \"Content-Length\";\n+static const char last_modified[] = \"Last-Modified\";\n+\n+static void format_write(int fd, const char *fmt, ...)\n+{\n+\tstatic char buffer[1024];\n+\n+\tva_list args;\n+\tunsigned n;\n+\n+\tva_start(args, fmt);\n+\tn = vsnprintf(buffer, sizeof(buffer), fmt, args);\n+\tva_end(args);\n+\tif (n >= sizeof(buffer))\n+\t\tdie(\"protocol error: impossibly long line\");\n+\n+\tsafe_write(fd, buffer, n);\n+}\n+\n+static void http_status(unsigned code, const char *msg)\n+{\n+\tformat_write(1, \"Status: %u %s\\r\\n\", code, msg);\n+}\n+\n+static void hdr_str(const char *name, const char *value)\n+{\n+\tformat_write(1, \"%s: %s\\r\\n\", name, value);\n+}\n+\n+static void hdr_int(const char *name, size_t value)\n+{\n+\tformat_write(1, \"%s: %\" PRIuMAX \"\\r\\n\", name, value);\n+}\n+\n+static void hdr_date(const char *name, unsigned long when)\n+{\n+\tconst char *value = show_date(when, 0, DATE_RFC2822);\n+\thdr_str(name, value);\n+}\n+\n+static void hdr_nocache(void)\n+{\n+\thdr_str(\"Expires\", \"Fri, 01 Jan 1980 00:00:00 GMT\");\n+\thdr_str(\"Pragma\", \"no-cache\");\n+\thdr_str(\"Cache-Control\", \"no-cache, max-age=0, must-revalidate\");\n+}\n+\n+static void hdr_cache_forever(void)\n+{\n+\tunsigned long now = time(NULL);\n+\thdr_date(\"Date\", now);\n+\thdr_date(\"Expires\", now + 31536000);\n+\thdr_str(\"Cache-Control\", \"public, max-age=31536000\");\n+}\n+\n+static void end_headers(void)\n+{\n+\tsafe_write(1, \"\\r\\n\", 2);\n+}\n+\n+static NORETURN void not_found(const char *err, ...)\n+{\n+\tva_list params;\n+\n+\thttp_status(404, \"Not Found\");\n+\thdr_nocache();\n+\tend_headers();\n+\n+\tva_start(params, err);\n+\tif (err && *err)\n+\t\tvfprintf(stderr, err, params);\n+\tva_end(params);\n+\texit(0);\n+}\n+\n+static void send_strbuf(const char *type, struct strbuf *buf)\n+{\n+\thdr_int(content_length, buf->len);\n+\thdr_str(content_type, type);\n+\tend_headers();\n+\tsafe_write(1, buf->buf, buf->len);\n+}\n+\n+static void send_file(const char *the_type, const char *name)\n+{\n+\tconst char *p = git_path(\"%s\", name);\n+\tsize_t buf_alloc = 8192;\n+\tchar *buf = xmalloc(buf_alloc);\n+\tint fd;\n+\tstruct stat sb;\n+\tsize_t size;\n+\n+\tfd = open(p, O_RDONLY);\n+\tif (fd < 0)\n+\t\tnot_found(\"Cannot open '%s': %s\", p, strerror(errno));\n+\tif (fstat(fd, &sb) < 0)\n+\t\tdie_errno(\"Cannot stat '%s'\", p);\n+\n+\tsize = xsize_t(sb.st_size);\n+\n+\thdr_int(content_length, size);\n+\thdr_str(content_type, the_type);\n+\thdr_date(last_modified, sb.st_mtime);\n+\tend_headers();\n+\n+\twhile (size) {\n+\t\tssize_t n = xread(fd, buf, buf_alloc);\n+\t\tif (n < 0)\n+\t\t\tdie_errno(\"Cannot read '%s'\", p);\n+\t\tif (!n)\n+\t\t\tbreak;\n+\t\tsafe_write(1, buf, n);\n+\t}\n+\tclose(fd);\n+\tfree(buf);\n+}\n+\n+static void get_text_file(char *name)\n+{\n+\thdr_nocache();\n+\tsend_file(\"text/plain\", name);\n+}\n+\n+static void get_loose_object(char *name)\n+{\n+\thdr_cache_forever();\n+\tsend_file(\"application/x-git-loose-object\", name);\n+}\n+\n+static void get_pack_file(char *name)\n+{\n+\thdr_cache_forever();\n+\tsend_file(\"application/x-git-packed-objects\", name);\n+}\n+\n+static void get_idx_file(char *name)\n+{\n+\thdr_cache_forever();\n+\tsend_file(\"application/x-git-packed-objects-toc\", name);\n+}\n+\n+static int show_text_ref(const char *name, const unsigned char *sha1,\n+\tint flag, void *cb_data)\n+{\n+\tstruct strbuf *buf = cb_data;\n+\tstruct object *o = parse_object(sha1);\n+\tif (!o)\n+\t\treturn 0;\n+\n+\tstrbuf_addf(buf, \"%s\\t%s\\n\", sha1_to_hex(sha1), name);\n+\tif (o->type == OBJ_TAG) {\n+\t\to = deref_tag(o, name, 0);\n+\t\tif (!o)\n+\t\t\treturn 0;\n+\t\tstrbuf_addf(buf, \"%s\\t%s^{}\\n\", sha1_to_hex(o->sha1), name);\n+\t}\n+\treturn 0;\n+}\n+\n+static void get_info_refs(char *arg)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tfor_each_ref(show_text_ref, &buf);\n+\thdr_nocache();\n+\tsend_strbuf(\"text/plain\", &buf);\n+\tstrbuf_release(&buf);\n+}\n+\n+static void get_info_packs(char *arg)\n+{\n+\tsize_t objdirlen = strlen(get_object_directory());\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\tsize_t cnt = 0;\n+\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif (p->pack_local)\n+\t\t\tcnt++;\n+\t}\n+\n+\tstrbuf_grow(&buf, cnt * 53 + 2);\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif (p->pack_local)\n+\t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n+\t}\n+\tstrbuf_addch(&buf, '\\n');\n+\n+\thdr_nocache();\n+\tsend_strbuf(\"text/plain; charset=utf-8\", &buf);\n+\tstrbuf_release(&buf);\n+}\n+\n+static NORETURN void die_webcgi(const char *err, va_list params)\n+{\n+\tchar buffer[1000];\n+\n+\thttp_status(500, \"Internal Server Error\");\n+\thdr_nocache();\n+\tend_headers();\n+\n+\tvsnprintf(buffer, sizeof(buffer), err, params);\n+\tfprintf(stderr, \"fatal: %s\\n\", buffer);\n+\texit(0);\n+}\n+\n+static struct service_cmd {\n+\tconst char *method;\n+\tconst char *pattern;\n+\tvoid (*imp)(char *);\n+} services[] = {\n+\t{\"GET\", \"/HEAD$\", get_text_file},\n+\t{\"GET\", \"/info/refs$\", get_info_refs},\n+\t{\"GET\", \"/objects/info/alternates$\", get_text_file},\n+\t{\"GET\", \"/objects/info/http-alternates$\", get_text_file},\n+\t{\"GET\", \"/objects/info/packs$\", get_info_packs},\n+\t{\"GET\", \"/objects/info/[^/]*$\", get_text_file},\n+\t{\"GET\", \"/objects/[0-9a-f]{2}/[0-9a-f]{38}$\", get_loose_object},\n+\t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.pack$\", get_pack_file},\n+\t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.idx$\", get_idx_file}\n+};\n+\n+int main(int argc, char **argv)\n+{\n+\tchar *method = getenv(\"REQUEST_METHOD\");\n+\tchar *dir = getenv(\"PATH_TRANSLATED\");\n+\tstruct service_cmd *cmd = NULL;\n+\tchar *cmd_arg = NULL;\n+\tint i;\n+\n+\tgit_extract_argv0_path(argv[0]);\n+\tset_die_routine(die_webcgi);\n+\n+\tif (!method)\n+\t\tdie(\"No REQUEST_METHOD from server\");\n+\tif (!strcmp(method, \"HEAD\"))\n+\t\tmethod = \"GET\";\n+\tif (!dir)\n+\t\tdie(\"No PATH_TRANSLATED from server\");\n+\n+\tfor (i = 0; i < ARRAY_SIZE(services); i++) {\n+\t\tstruct service_cmd *c = &services[i];\n+\t\tregex_t re;\n+\t\tregmatch_t out[1];\n+\n+\t\tif (regcomp(&re, c->pattern, REG_EXTENDED))\n+\t\t\tdie(\"Bogus regex in service table: %s\", c->pattern);\n+\t\tif (!regexec(&re, dir, 1, out, 0)) {\n+\t\t\tsize_t n = out[0].rm_eo - out[0].rm_so;\n+\n+\t\t\tif (strcmp(method, c->method)) {\n+\t\t\t\tconst char *proto = getenv(\"SERVER_PROTOCOL\");\n+\t\t\t\tif (proto && !strcmp(proto, \"HTTP/1.1\"))\n+\t\t\t\t\thttp_status(405, \"Method Not Allowed\");\n+\t\t\t\telse\n+\t\t\t\t\thttp_status(400, \"Bad Request\");\n+\t\t\t\thdr_nocache();\n+\t\t\t\tend_headers();\n+\t\t\t\treturn 0;\n+\t\t\t}\n+\n+\t\t\tcmd = c;\n+\t\t\tcmd_arg = xmalloc(n);\n+\t\t\tstrncpy(cmd_arg, dir + out[0].rm_so + 1, n);\n+\t\t\tcmd_arg[n] = '\\0';\n+\t\t\tdir[out[0].rm_so] = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tregfree(&re);\n+\t}\n+\n+\tif (!cmd)\n+\t\tnot_found(\"Request not supported: '%s'\", dir);\n+\n+\tsetup_path();\n+\tif (!enter_repo(dir, 0))\n+\t\tnot_found(\"Not a git repository: '%s'\", dir);\n+\n+\tcmd->imp(cmd_arg);\n+\treturn 0;\n+}\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126196","messageId":"1256774448-7625-15-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 14/26] Add stateless RPC options to upload-pack, receive-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:36Z","receivedAt":"2009-10-29T00:00:36Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When --stateless-rpc is passed as a command line parameter to\nupload-pack or receive-pack the programs now assume they may\nperform only a single read-write cycle with stdin and stdout.\nThis fits with the HTTP POST request processing model where a\nprogram may read the request, write a response, and must exit.\n\nWhen --advertise-refs is passed as a command line parameter only\nthe initial ref advertisement is output, and the program exits\nimmediately. This fits with the HTTP GET request model, where\nno request content is received but a response must be produced.\n\nHTTP headers and/or environment are not processed here, but\ninstead are assumed to be handled by the program invoking\neither service backend.\n\nSigned-off-by: Shawn O. Pearce \n---\n builtin-receive-pack.c | 26 ++++++++++++++++++++------\n upload-pack.c | 40 ++++++++++++++++++++++++++++++++++++----\n 2 files changed, 56 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin-receive-pack.c b/builtin-receive-pack.c\nindex b771fe9..70ff8c5 100644\n--- a/builtin-receive-pack.c\n+++ b/builtin-receive-pack.c\n@@ -615,6 +615,8 @@ static void add_alternate_refs(void)\n \n int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n {\n+\tint advertise_refs = 0;\n+\tint stateless_rpc = 0;\n \tint i;\n \tchar *dir = NULL;\n \n@@ -623,7 +625,15 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tconst char *arg = *argv++;\n \n \t\tif (*arg == '-') {\n-\t\t\t/* Do flag handling here */\n+\t\t\tif (!strcmp(arg, \"--advertise-refs\")) {\n+\t\t\t\tadvertise_refs = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!strcmp(arg, \"--stateless-rpc\")) {\n+\t\t\t\tstateless_rpc = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tusage(receive_pack_usage);\n \t\t}\n \t\tif (dir)\n@@ -652,12 +662,16 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\t\" report-status delete-refs ofs-delta \" :\n \t\t\" report-status delete-refs \";\n \n-\tadd_alternate_refs();\n-\twrite_head_info();\n-\tclear_extra_refs();\n+\tif (advertise_refs || !stateless_rpc) {\n+\t\tadd_alternate_refs();\n+\t\twrite_head_info();\n+\t\tclear_extra_refs();\n \n-\t/* EOF */\n-\tpacket_flush(1);\n+\t\t/* EOF */\n+\t\tpacket_flush(1);\n+\t}\n+\tif (advertise_refs)\n+\t\treturn 0;\n \n \tread_head_info();\n \tif (commands) {\ndiff --git a/upload-pack.c b/upload-pack.c\nindex f1dc3a3..70badcf 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -39,6 +39,8 @@ static unsigned int timeout;\n */\n static int use_sideband;\n static int debug_fd;\n+static int advertise_refs;\n+static int stateless_rpc;\n \n static void reset_timeout(void)\n {\n@@ -509,6 +511,8 @@ static int get_common_commits(void)\n \t\tif (!len) {\n \t\t\tif (have_obj.nr == 0 || multi_ack)\n \t\t\t\tpacket_write(1, \"NAK\\n\");\n+\t\t\tif (stateless_rpc)\n+\t\t\t\texit(0);\n \t\t\tcontinue;\n \t\t}\n \t\tstrip(line, len);\n@@ -710,12 +714,32 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo\n \treturn 0;\n }\n \n+static int mark_our_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)\n+{\n+\tstruct object *o = parse_object(sha1);\n+\tif (!o)\n+\t\tdie(\"git upload-pack: cannot find object %s:\", sha1_to_hex(sha1));\n+\tif (!(o->flags & OUR_REF)) {\n+\t\to->flags |= OUR_REF;\n+\t\tnr_our_refs++;\n+\t}\n+\treturn 0;\n+}\n+\n static void upload_pack(void)\n {\n-\treset_timeout();\n-\thead_ref(send_ref, NULL);\n-\tfor_each_ref(send_ref, NULL);\n-\tpacket_flush(1);\n+\tif (advertise_refs || !stateless_rpc) {\n+\t\treset_timeout();\n+\t\thead_ref(send_ref, NULL);\n+\t\tfor_each_ref(send_ref, NULL);\n+\t\tpacket_flush(1);\n+\t} else {\n+\t\thead_ref(mark_our_ref, NULL);\n+\t\tfor_each_ref(mark_our_ref, NULL);\n+\t}\n+\tif (advertise_refs)\n+\t\treturn;\n+\n \treceive_needs();\n \tif (want_obj.nr) {\n \t\tget_common_commits();\n@@ -737,6 +761,14 @@ int main(int argc, char **argv)\n \n \t\tif (arg[0] != '-')\n \t\t\tbreak;\n+\t\tif (!strcmp(arg, \"--advertise-refs\")) {\n+\t\t\tadvertise_refs = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--stateless-rpc\")) {\n+\t\t\tstateless_rpc = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--strict\")) {\n \t\t\tstrict = 1;\n \t\t\tcontinue;\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126198","messageId":"1256774448-7625-16-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 15/26] Smart fetch and push over HTTP: server side","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:37Z","receivedAt":"2009-10-29T00:00:37Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Requests for $GIT_URL/git-receive-pack and $GIT_URL/git-upload-pack\nare forwarded to the corresponding backend process by directly\nexecuting it and leaving stdin and stdout connected to the invoking\nweb server. Prior to starting the backend process the HTTP response\nheaders are sent, thereby freeing the backend from needing to know\nabout the HTTP protocol.\n\nRequests that are encoded with Content-Encoding: gzip are\nautomatically inflated before being streamed into the backend.\nThis is primarily useful for the git-upload-pack backend, which\nreceives highly repetitive text data from clients that easily\ncompresses to 50% of its original size.\n\nSigned-off-by: Shawn O. Pearce \n---\n Documentation/git-http-backend.txt | 39 +++++-\n http-backend.c | 324 +++++++++++++++++++++++++++++++++++-\n 2 files changed, 359 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt\nindex 867675f..022a243 100644\n--- a/Documentation/git-http-backend.txt\n+++ b/Documentation/git-http-backend.txt\n@@ -22,6 +22,23 @@ By default, only the `upload-pack` service is enabled, which serves\n This is ideally suited for read-only updates, i.e., pulling from\n git repositories.\n \n+SERVICES\n+--------\n+These services can be enabled/disabled using the per-repository\n+configuration file:\n+\n+http.uploadpack::\n+\tThis serves 'git-fetch-pack' and 'git-ls-remote' clients.\n+\tIt is enabled by default, but a repository can disable it\n+\tby setting this configuration item to `false`.\n+\n+http.receivepack::\n+\tThis serves 'git-send-pack' clients, allowing push. It is\n+\tdisabled by default for anonymous users, and enabled by\n+\tdefault for users authenticated by the web server. It can be\n+\tdisabled by setting this item to `false`, or enabled for all\n+\tusers, including anonymous users, by setting it to `true`.\n+\n URL TRANSLATION\n ---------------\n 'git-http-backend' relies on the invoking web server to perform\n@@ -49,7 +66,19 @@ ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/git/\n \n ----------------------------------------------------------------\n +\n-To require authentication for reads, use a Directory\n+To enable anonymous read access but authenticated write access,\n+require authorization with a LocationMatch directive:\n++\n+----------------------------------------------------------------\n+\n+\tAuthType Basic\n+\tAuthName \"Git Access\"\n+\tRequire group committers\n+\t...\n+\n+----------------------------------------------------------------\n++\n+To require authentication for both reads and writes, use a Directory\n directive around the repository, or one of its parent directories:\n +\n ----------------------------------------------------------------\n@@ -92,6 +121,14 @@ by the invoking web server, including:\n * QUERY_STRING\n * REQUEST_METHOD\n \n+The backend process sets GIT_COMMITTER_NAME to '$REMOTE_USER' and\n+GIT_COMMITTER_EMAIL to '$\\{REMOTE_USER}@http.$\\{REMOTE_ADDR\\}',\n+ensuring that any reflogs created by 'git-receive-pack' contain some\n+identifying information of the remote user who performed the push.\n+\n+All CGI environment variables are available to each of the hooks\n+invoked by the 'git-receive-pack'.\n+\n Author\n ------\n Written by Shawn O. Pearce .\ndiff --git a/http-backend.c b/http-backend.c\nindex 374f60d..67030b5 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -4,11 +4,109 @@\n #include \"object.h\"\n #include \"tag.h\"\n #include \"exec_cmd.h\"\n+#include \"run-command.h\"\n+#include \"string-list.h\"\n \n static const char content_type[] = \"Content-Type\";\n static const char content_length[] = \"Content-Length\";\n static const char last_modified[] = \"Last-Modified\";\n \n+static struct string_list *query_params;\n+\n+struct rpc_service {\n+\tconst char *name;\n+\tconst char *config_name;\n+\tsigned enabled : 2;\n+};\n+\n+static struct rpc_service rpc_service[] = {\n+\t{ \"upload-pack\", \"uploadpack\", 1 },\n+\t{ \"receive-pack\", \"receivepack\", -1 },\n+};\n+\n+static int decode_char(const char *q)\n+{\n+\tint i;\n+\tunsigned char val = 0;\n+\tfor (i = 0; i < 2; i++) {\n+\t\tunsigned char c = *q++;\n+\t\tval <<= 4;\n+\t\tif (c >= '0' && c <= '9')\n+\t\t\tval += c - '0';\n+\t\telse if (c >= 'a' && c <= 'f')\n+\t\t\tval += c - 'a' + 10;\n+\t\telse if (c >= 'A' && c <= 'F')\n+\t\t\tval += c - 'A' + 10;\n+\t\telse\n+\t\t\treturn -1;\n+\t}\n+\treturn val;\n+}\n+\n+static char *decode_parameter(const char **query, int is_name)\n+{\n+\tconst char *q = *query;\n+\tstruct strbuf out;\n+\n+\tstrbuf_init(&out, 16);\n+\tdo {\n+\t\tunsigned char c = *q;\n+\n+\t\tif (!c)\n+\t\t\tbreak;\n+\t\tif (c == '&' || (is_name && c == '=')) {\n+\t\t\tq++;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (c == '%') {\n+\t\t\tint val = decode_char(q + 1);\n+\t\t\tif (0 <= val) {\n+\t\t\t\tstrbuf_addch(&out, val);\n+\t\t\t\tq += 3;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (c == '+')\n+\t\t\tstrbuf_addch(&out, ' ');\n+\t\telse\n+\t\t\tstrbuf_addch(&out, c);\n+\t\tq++;\n+\t} while (1);\n+\t*query = q;\n+\treturn strbuf_detach(&out, NULL);\n+}\n+\n+static struct string_list *get_parameters(void)\n+{\n+\tif (!query_params) {\n+\t\tconst char *query = getenv(\"QUERY_STRING\");\n+\n+\t\tquery_params = xcalloc(1, sizeof(*query_params));\n+\t\twhile (query && *query) {\n+\t\t\tchar *name = decode_parameter(&query, 1);\n+\t\t\tchar *value = decode_parameter(&query, 0);\n+\t\t\tstruct string_list_item *i;\n+\n+\t\t\ti = string_list_lookup(name, query_params);\n+\t\t\tif (!i)\n+\t\t\t\ti = string_list_insert(name, query_params);\n+\t\t\telse\n+\t\t\t\tfree(i->util);\n+\t\t\ti->util = value;\n+\t\t}\n+\t}\n+\treturn query_params;\n+}\n+\n+static const char *get_parameter(const char *name)\n+{\n+\tstruct string_list_item *i;\n+\ti = string_list_lookup(name, get_parameters());\n+\treturn i ? i->util : NULL;\n+}\n+\n static void format_write(int fd, const char *fmt, ...)\n {\n \tstatic char buffer[1024];\n@@ -81,6 +179,21 @@ static NORETURN void not_found(const char *err, ...)\n \texit(0);\n }\n \n+static NORETURN void forbidden(const char *err, ...)\n+{\n+\tva_list params;\n+\n+\thttp_status(403, \"Forbidden\");\n+\thdr_nocache();\n+\tend_headers();\n+\n+\tva_start(params, err);\n+\tif (err && *err)\n+\t\tvfprintf(stderr, err, params);\n+\tva_end(params);\n+\texit(0);\n+}\n+\n static void send_strbuf(const char *type, struct strbuf *buf)\n {\n \thdr_int(content_length, buf->len);\n@@ -147,6 +260,145 @@ static void get_idx_file(char *name)\n \tsend_file(\"application/x-git-packed-objects-toc\", name);\n }\n \n+static int http_config(const char *var, const char *value, void *cb)\n+{\n+\tstruct rpc_service *svc = cb;\n+\n+\tif (!prefixcmp(var, \"http.\") &&\n+\t !strcmp(var + 5, svc->config_name)) {\n+\t\tsvc->enabled = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\t/* we are not interested in parsing any other configuration here */\n+\treturn 0;\n+}\n+\n+static struct rpc_service *select_service(const char *name)\n+{\n+\tstruct rpc_service *svc = NULL;\n+\tint i;\n+\n+\tif (prefixcmp(name, \"git-\"))\n+\t\tforbidden(\"Unsupported service: '%s'\", name);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(rpc_service); i++) {\n+\t\tstruct rpc_service *s = &rpc_service[i];\n+\t\tif (!strcmp(s->name, name + 4)) {\n+\t\t\tsvc = s;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (!svc)\n+\t\tforbidden(\"Unsupported service: '%s'\", name);\n+\n+\tgit_config(http_config, svc);\n+\tif (svc->enabled < 0) {\n+\t\tconst char *user = getenv(\"REMOTE_USER\");\n+\t\tsvc->enabled = (user && *user) ? 1 : 0;\n+\t}\n+\tif (!svc->enabled)\n+\t\tforbidden(\"Service not enabled: '%s'\", svc->name);\n+\treturn svc;\n+}\n+\n+static void inflate_request(const char *prog_name, int out)\n+{\n+\tz_stream stream;\n+\tunsigned char in_buf[8192];\n+\tunsigned char out_buf[8192];\n+\tunsigned long cnt = 0;\n+\tint ret;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tret = inflateInit2(&stream, (15 + 16));\n+\tif (ret != Z_OK)\n+\t\tdie(\"cannot start zlib inflater, zlib err %d\", ret);\n+\n+\twhile (1) {\n+\t\tssize_t n = xread(0, in_buf, sizeof(in_buf));\n+\t\tif (n <= 0)\n+\t\t\tdie(\"request ended in the middle of the gzip stream\");\n+\n+\t\tstream.next_in = in_buf;\n+\t\tstream.avail_in = n;\n+\n+\t\twhile (0 < stream.avail_in) {\n+\t\t\tint ret;\n+\n+\t\t\tstream.next_out = out_buf;\n+\t\t\tstream.avail_out = sizeof(out_buf);\n+\n+\t\t\tret = inflate(&stream, Z_NO_FLUSH);\n+\t\t\tif (ret != Z_OK && ret != Z_STREAM_END)\n+\t\t\t\tdie(\"zlib error inflating request, result %d\", ret);\n+\n+\t\t\tn = stream.total_out - cnt;\n+\t\t\tif (write_in_full(out, out_buf, n) != n)\n+\t\t\t\tdie(\"%s aborted reading request\", prog_name);\n+\t\t\tcnt += n;\n+\n+\t\t\tif (ret == Z_STREAM_END)\n+\t\t\t\tgoto done;\n+\t\t}\n+\t}\n+\n+done:\n+\tinflateEnd(&stream);\n+\tclose(out);\n+}\n+\n+static void run_service(const char **argv)\n+{\n+\tconst char *encoding = getenv(\"HTTP_CONTENT_ENCODING\");\n+\tconst char *user = getenv(\"REMOTE_USER\");\n+\tconst char *host = getenv(\"REMOTE_ADDR\");\n+\tchar *env[3];\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint gzipped_request = 0;\n+\tstruct child_process cld;\n+\n+\tif (encoding && !strcmp(encoding, \"gzip\"))\n+\t\tgzipped_request = 1;\n+\telse if (encoding && !strcmp(encoding, \"x-gzip\"))\n+\t\tgzipped_request = 1;\n+\n+\tif (!user || !*user)\n+\t\tuser = \"anonymous\";\n+\tif (!host || !*host)\n+\t\thost = \"(none)\";\n+\n+\tmemset(&env, 0, sizeof(env));\n+\tstrbuf_addf(&buf, \"GIT_COMMITTER_NAME=%s\", user);\n+\tenv[0] = strbuf_detach(&buf, NULL);\n+\n+\tstrbuf_addf(&buf, \"GIT_COMMITTER_EMAIL=%s@http.%s\", user, host);\n+\tenv[1] = strbuf_detach(&buf, NULL);\n+\tenv[2] = NULL;\n+\n+\tmemset(&cld, 0, sizeof(cld));\n+\tcld.argv = argv;\n+\tcld.env = (const char *const *)env;\n+\tif (gzipped_request)\n+\t\tcld.in = -1;\n+\tcld.git_cmd = 1;\n+\tif (start_command(&cld))\n+\t\texit(1);\n+\n+\tclose(1);\n+\tif (gzipped_request)\n+\t\tinflate_request(argv[0], cld.in);\n+\telse\n+\t\tclose(0);\n+\n+\tif (finish_command(&cld))\n+\t\texit(1);\n+\tfree(env[0]);\n+\tfree(env[1]);\n+\tstrbuf_release(&buf);\n+}\n+\n static int show_text_ref(const char *name, const unsigned char *sha1,\n \tint flag, void *cb_data)\n {\n@@ -167,11 +419,32 @@ static int show_text_ref(const char *name, const unsigned char *sha1,\n \n static void get_info_refs(char *arg)\n {\n+\tconst char *service_name = get_parameter(\"service\");\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tfor_each_ref(show_text_ref, &buf);\n \thdr_nocache();\n-\tsend_strbuf(\"text/plain\", &buf);\n+\n+\tif (service_name) {\n+\t\tconst char *argv[] = {NULL /* service name */,\n+\t\t\t\"--stateless-rpc\", \"--advertise-refs\",\n+\t\t\t\".\", NULL};\n+\t\tstruct rpc_service *svc = select_service(service_name);\n+\n+\t\tstrbuf_addf(&buf, \"application/x-git-%s-advertisement\",\n+\t\t\tsvc->name);\n+\t\thdr_str(content_type, buf.buf);\n+\t\tend_headers();\n+\n+\t\tpacket_write(1, \"# service=git-%s\\n\", svc->name);\n+\t\tpacket_flush(1);\n+\n+\t\targv[0] = svc->name;\n+\t\trun_service(argv);\n+\n+\t} else {\n+\t\tfor_each_ref(show_text_ref, &buf);\n+\t\tsend_strbuf(\"text/plain\", &buf);\n+\t}\n \tstrbuf_release(&buf);\n }\n \n@@ -200,6 +473,48 @@ static void get_info_packs(char *arg)\n \tstrbuf_release(&buf);\n }\n \n+static void check_content_type(const char *accepted_type)\n+{\n+\tconst char *actual_type = getenv(\"CONTENT_TYPE\");\n+\n+\tif (!actual_type)\n+\t\tactual_type = \"\";\n+\n+\tif (strcmp(actual_type, accepted_type)) {\n+\t\thttp_status(415, \"Unsupported Media Type\");\n+\t\thdr_nocache();\n+\t\tend_headers();\n+\t\tformat_write(1,\n+\t\t\t\"Expected POST with Content-Type '%s',\"\n+\t\t\t\" but received '%s' instead.\\n\",\n+\t\t\taccepted_type, actual_type);\n+\t\texit(0);\n+\t}\n+}\n+\n+static void service_rpc(char *service_name)\n+{\n+\tconst char *argv[] = {NULL, \"--stateless-rpc\", \".\", NULL};\n+\tstruct rpc_service *svc = select_service(service_name);\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addf(&buf, \"application/x-git-%s-request\", svc->name);\n+\tcheck_content_type(buf.buf);\n+\n+\thdr_nocache();\n+\n+\tstrbuf_reset(&buf);\n+\tstrbuf_addf(&buf, \"application/x-git-%s-result\", svc->name);\n+\thdr_str(content_type, buf.buf);\n+\n+\tend_headers();\n+\n+\targv[0] = svc->name;\n+\trun_service(argv);\n+\tstrbuf_release(&buf);\n+}\n+\n static NORETURN void die_webcgi(const char *err, va_list params)\n {\n \tchar buffer[1000];\n@@ -226,7 +541,10 @@ static struct service_cmd {\n \t{\"GET\", \"/objects/info/[^/]*$\", get_text_file},\n \t{\"GET\", \"/objects/[0-9a-f]{2}/[0-9a-f]{38}$\", get_loose_object},\n \t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.pack$\", get_pack_file},\n-\t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.idx$\", get_idx_file}\n+\t{\"GET\", \"/objects/pack/pack-[0-9a-f]{40}\\\\.idx$\", get_idx_file},\n+\n+\t{\"POST\", \"/git-upload-pack$\", service_rpc},\n+\t{\"POST\", \"/git-receive-pack$\", service_rpc}\n };\n \n int main(int argc, char **argv)\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126202","messageId":"1256774448-7625-17-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 16/26] http-backend: add GIT_PROJECT_ROOT environment var","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:38Z","receivedAt":"2009-10-29T00:00:38Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"From: Mark Lodato \n\nAdd a new environment variable, GIT_PROJECT_ROOT, to override the\nmethod of using PATH_TRANSLATED to find the git repository on disk.\nThis makes it much easier to configure the web server, especially when\nthe web server's DocumentRoot does not contain the git repositories,\nwhich is the usual case.\n\nSigned-off-by: Mark Lodato \nSigned-off-by: Shawn O. Pearce \n---\n Documentation/git-http-backend.txt | 39 +++++++++++++++--------------------\n http-backend.c | 25 ++++++++++++++++++++--\n 2 files changed, 39 insertions(+), 25 deletions(-)\n\ndiff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt\nindex 022a243..99dbbfb 100644\n--- a/Documentation/git-http-backend.txt\n+++ b/Documentation/git-http-backend.txt\n@@ -41,29 +41,24 @@ http.receivepack::\n \n URL TRANSLATION\n ---------------\n-'git-http-backend' relies on the invoking web server to perform\n-URL to path translation, and store the repository path into the\n-PATH_TRANSLATED environment variable. Most web servers will do\n-this translation automatically, resolving the suffix after the\n-CGI name relative to the server's document root.\n+To determine the location of the repository on disk, 'git-http-backend'\n+concatenates the environment variables PATH_INFO, which is set\n+automatically by the web server, and GIT_PROJECT_ROOT, which must be set\n+manually in the web server configuration. If GIT_PROJECT_ROOT is not\n+set, 'git-http-backend' reads PATH_TRANSLATED, which is also set\n+automatically by the web server.\n \n EXAMPLES\n --------\n \n Apache 2.x::\n-\tTo serve all Git repositories contained within the '/git/'\n-\tsubdirectory of the DocumentRoot, ensure mod_cgi and\n-\tmod_alias are enabled, and create a ScriptAlias to the CGI:\n+\tEnsure mod_cgi, mod_alias, and mod_env are enabled, set\n+\tGIT_PROJECT_ROOT (or DocumentRoot) appropriately, and\n+\tcreate a ScriptAlias to the CGI:\n +\n ----------------------------------------------------------------\n-ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/git/\n-\n-\n-\tOptions None\n-\n-\n-\tOptions ExecCGI\n-\n+SetEnv GIT_PROJECT_ROOT /var/www/git\n+ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/\n ----------------------------------------------------------------\n +\n To enable anonymous read access but authenticated write access,\n@@ -78,16 +73,16 @@ require authorization with a LocationMatch directive:\n \n ----------------------------------------------------------------\n +\n-To require authentication for both reads and writes, use a Directory\n+To require authentication for both reads and writes, use a Location\n directive around the repository, or one of its parent directories:\n +\n ----------------------------------------------------------------\n-\n+\n \tAuthType Basic\n \tAuthName \"Private Git Access\"\n \tRequire group committers\n \t...\n-\n+\n ----------------------------------------------------------------\n \n Accelerated static Apache 2.x::\n@@ -97,9 +92,9 @@ Accelerated static Apache 2.x::\n \tfile contents from the file system directly to the network:\n +\n ----------------------------------------------------------------\n-DocumentRoot /var/www\n+SetEnv GIT_PROJECT_ROOT /var/www/git\n \n-ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/git/\n+ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/\n Alias /git_static/ /var/www/git/\n \n RewriteEngine on\n@@ -114,7 +109,7 @@ ENVIRONMENT\n 'git-http-backend' relies upon the CGI environment variables set\n by the invoking web server, including:\n \n-* PATH_TRANSLATED\n+* PATH_INFO (if GIT_PROJECT_ROOT is set, otherwise PATH_TRANSLATED)\n * REMOTE_USER\n * REMOTE_ADDR\n * CONTENT_TYPE\ndiff --git a/http-backend.c b/http-backend.c\nindex 67030b5..8e5c0a2 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -528,6 +528,26 @@ static NORETURN void die_webcgi(const char *err, va_list params)\n \texit(0);\n }\n \n+static char* getdir(void)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tchar *pathinfo = getenv(\"PATH_INFO\");\n+\tchar *root = getenv(\"GIT_PROJECT_ROOT\");\n+\tchar *path = getenv(\"PATH_TRANSLATED\");\n+\n+\tif (root && *root) {\n+\t\tif (!pathinfo || !*pathinfo)\n+\t\t\tdie(\"GIT_PROJECT_ROOT is set but PATH_INFO is not\");\n+\t\tstrbuf_addstr(&buf, root);\n+\t\tstrbuf_addstr(&buf, pathinfo);\n+\t\treturn strbuf_detach(&buf, NULL);\n+\t} else if (path && *path) {\n+\t\treturn xstrdup(path);\n+\t} else\n+\t\tdie(\"No GIT_PROJECT_ROOT or PATH_TRANSLATED from server\");\n+\treturn NULL;\n+}\n+\n static struct service_cmd {\n \tconst char *method;\n \tconst char *pattern;\n@@ -550,7 +570,7 @@ static struct service_cmd {\n int main(int argc, char **argv)\n {\n \tchar *method = getenv(\"REQUEST_METHOD\");\n-\tchar *dir = getenv(\"PATH_TRANSLATED\");\n+\tchar *dir;\n \tstruct service_cmd *cmd = NULL;\n \tchar *cmd_arg = NULL;\n \tint i;\n@@ -562,8 +582,7 @@ int main(int argc, char **argv)\n \t\tdie(\"No REQUEST_METHOD from server\");\n \tif (!strcmp(method, \"HEAD\"))\n \t\tmethod = \"GET\";\n-\tif (!dir)\n-\t\tdie(\"No PATH_TRANSLATED from server\");\n+\tdir = getdir();\n \n \tfor (i = 0; i < ARRAY_SIZE(services); i++) {\n \t\tstruct service_cmd *c = &services[i];\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126204","messageId":"1256774448-7625-18-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 17/26] http-backend: reword some documentation","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:39Z","receivedAt":"2009-10-29T00:00:39Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"From: Mark Lodato \n\nClarify some of the git-http-backend documentation, particularly:\n\n* In the Description, state that smart/dumb HTTP fetch and smart HTTP\n push are supported, state that authenticated clients allow push, and\n remove the note that this is only suited for read-only updates.\n\n* At the start of Examples, state explicitly what URL is mapping to what\n location on disk.\n\nSigned-off-by: Mark Lodato \nSigned-off-by: Shawn O. Pearce \n---\n Documentation/git-http-backend.txt | 12 ++++++++----\n 1 files changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt\nindex 99dbbfb..0b5e951 100644\n--- a/Documentation/git-http-backend.txt\n+++ b/Documentation/git-http-backend.txt\n@@ -14,13 +14,15 @@ DESCRIPTION\n -----------\n A simple CGI program to serve the contents of a Git repository to Git\n clients accessing the repository over http:// and https:// protocols.\n+The program supports clients fetching using both the smart HTTP protcol\n+and the backwards-compatible dumb HTTP protocol, as well as clients\n+pushing using the smart HTTP protocol.\n \n By default, only the `upload-pack` service is enabled, which serves\n 'git-fetch-pack' and 'git-ls-remote' clients, which are invoked from\n-'git-fetch', 'git-pull', and 'git-clone'.\n-\n-This is ideally suited for read-only updates, i.e., pulling from\n-git repositories.\n+'git-fetch', 'git-pull', and 'git-clone'. If the client is authenticated,\n+the `receive-pack` service is enabled, which serves 'git-send-pack'\n+clients, which is invoked from 'git-push'.\n \n SERVICES\n --------\n@@ -50,6 +52,8 @@ automatically by the web server.\n \n EXAMPLES\n --------\n+All of the following examples map 'http://$hostname/git/foo/bar.git'\n+to '/var/www/git/foo/bar.git'.\n \n Apache 2.x::\n \tEnsure mod_cgi, mod_alias, and mod_env are enabled, set\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126205","messageId":"1256774448-7625-19-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 18/26] http-backend: use mod_alias instead of mod_rewrite","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:40Z","receivedAt":"2009-10-29T00:00:40Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"From: Mark Lodato \n\nIn the git-http-backend documentation, use mod_alias exlusively, instead\nof using a combination of mod_alias and mod_rewrite. This makes the\nexample slightly shorted and a bit more clear.\n\nSigned-off-by: Mark Lodato \nSigned-off-by: Shawn O. Pearce \n---\n Documentation/git-http-backend.txt | 10 +++-------\n 1 files changed, 3 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt\nindex 0b5e951..e67519d 100644\n--- a/Documentation/git-http-backend.txt\n+++ b/Documentation/git-http-backend.txt\n@@ -98,13 +98,9 @@ Accelerated static Apache 2.x::\n ----------------------------------------------------------------\n SetEnv GIT_PROJECT_ROOT /var/www/git\n \n-ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/\n-Alias /git_static/ /var/www/git/\n-\n-RewriteEngine on\n-RewriteRule ^/git/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /git_static/$1 [PT]\n-RewriteRule ^/git/(.*/objects/pack/pack-[0-9a-f]{40}.pack)$ /git_static/$1 [PT]\n-RewriteRule ^/git/(.*/objects/pack/pack-[0-9a-f]{40}.idx)$ /git_static/$1 [PT]\n+AliasMatch ^/git/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /var/www/git/$1\n+AliasMatch ^/git/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /var/www/git/$1\n+ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/\n ----------------------------------------------------------------\n \n \n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126200","messageId":"1256774448-7625-20-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 19/26] http-backend: add example for gitweb on same URL","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:41Z","receivedAt":"2009-10-29T00:00:41Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"From: Mark Lodato \n\nIn the git-http-backend documentation, add an example of how to set up\ngitweb and git-http-backend on the same URL by using a series of\nmod_alias commands.\n\nSigned-off-by: Mark Lodato \nSigned-off-by: Shawn O. Pearce \n---\n Documentation/git-http-backend.txt | 33 +++++++++++++++++++++++++++++++++\n 1 files changed, 33 insertions(+), 0 deletions(-)\n\ndiff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt\nindex e67519d..2989c9f 100644\n--- a/Documentation/git-http-backend.txt\n+++ b/Documentation/git-http-backend.txt\n@@ -88,6 +88,23 @@ directive around the repository, or one of its parent directories:\n \t...\n \n ----------------------------------------------------------------\n++\n+To serve gitweb at the same url, use a ScriptAliasMatch to only\n+those URLs that 'git-http-backend' can handle, and forward the\n+rest to gitweb:\n++\n+----------------------------------------------------------------\n+ScriptAliasMatch \\\n+\t\"(?x)^/git/(.*/(HEAD | \\\n+\t\t\tinfo/refs | \\\n+\t\t\tobjects/(info/[^/]+ | \\\n+\t\t\t\t [0-9a-f]{2}/[0-9a-f]{38} | \\\n+\t\t\t\t pack/pack-[0-9a-f]{40}\\.(pack|idx)) | \\\n+\t\t\tgit-(upload|receive)-pack))$\" \\\n+\t/usr/libexec/git-core/git-http-backend/$1\n+\n+ScriptAlias /git/ /var/www/cgi-bin/gitweb.cgi/\n+----------------------------------------------------------------\n \n Accelerated static Apache 2.x::\n \tSimilar to the above, but Apache can be used to return static\n@@ -102,6 +119,22 @@ AliasMatch ^/git/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /var/www/git/$1\n AliasMatch ^/git/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /var/www/git/$1\n ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/\n ----------------------------------------------------------------\n++\n+This can be combined with the gitweb configuration:\n++\n+----------------------------------------------------------------\n+SetEnv GIT_PROJECT_ROOT /var/www/git\n+\n+AliasMatch ^/git/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /var/www/git/$1\n+AliasMatch ^/git/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /var/www/git/$1\n+ScriptAliasMatch \\\n+\t\"(?x)^/git/(.*/(HEAD | \\\n+\t\t\tinfo/refs | \\\n+\t\t\tobjects/info/[^/]+ | \\\n+\t\t\tgit-(upload|receive)-pack))$\" \\\n+\t/usr/libexec/git-core/git-http-backend/$1\n+ScriptAlias /git/ /var/www/cgi-bin/gitweb.cgi/\n+----------------------------------------------------------------\n \n \n ENVIRONMENT\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126206","messageId":"1256774448-7625-21-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 20/26] http-backend: more explict LocationMatch","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:42Z","receivedAt":"2009-10-29T00:00:42Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"From: Mark Lodato \n\nIn the git-http-backend examples, only match git-receive-pack within\n/git/.\n\nSigned-off-by: Mark Lodato \nSigned-off-by: Shawn O. Pearce \n---\n Documentation/git-http-backend.txt | 2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-http-backend.txt b/Documentation/git-http-backend.txt\nindex 2989c9f..f17251a 100644\n--- a/Documentation/git-http-backend.txt\n+++ b/Documentation/git-http-backend.txt\n@@ -69,7 +69,7 @@ To enable anonymous read access but authenticated write access,\n require authorization with a LocationMatch directive:\n +\n ----------------------------------------------------------------\n-\n+\n \tAuthType Basic\n \tAuthName \"Git Access\"\n \tRequire group committers\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126201","messageId":"1256774448-7625-22-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 21/26] Discover refs via smart HTTP server when available","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:43Z","receivedAt":"2009-10-29T00:00:43Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Instead of loading the cached info/refs, try to use the smart HTTP\nversion when the server supports it. Since the smart variant is\nactually the pkt-line stream from the start of either upload-pack\nor receive-pack we need to parse these through get_remote_heads,\nwhich requires a background thread to feed its pipe.\n\nSigned-off-by: Shawn O. Pearce \nCC: Daniel Barkalow \n---\n remote-curl.c | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++-------\n 1 files changed, 131 insertions(+), 17 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 5c9dd97..3917d45 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -5,6 +5,7 @@\n #include \"http.h\"\n #include \"exec_cmd.h\"\n #include \"run-command.h\"\n+#include \"pkt-line.h\"\n \n static struct remote *remote;\n static const char *url;\n@@ -75,21 +76,46 @@ static int set_option(const char *name, const char *value)\n \t}\n }\n \n-static struct ref *get_refs(void)\n+struct discovery {\n+\tconst char *service;\n+\tchar *buf_alloc;\n+\tchar *buf;\n+\tsize_t len;\n+\tunsigned proto_git : 1;\n+};\n+static struct discovery *last_discovery;\n+\n+static void free_discovery(struct discovery *d)\n+{\n+\tif (d) {\n+\t\tif (d == last_discovery)\n+\t\t\tlast_discovery = NULL;\n+\t\tfree(d->buf_alloc);\n+\t\tfree(d);\n+\t}\n+}\n+\n+static struct discovery* discover_refs(const char *service)\n {\n \tstruct strbuf buffer = STRBUF_INIT;\n-\tchar *data, *start, *mid;\n-\tchar *ref_name;\n+\tstruct discovery *last = last_discovery;\n \tchar *refs_url;\n-\tint i = 0;\n-\tint http_ret;\n+\tint http_ret, is_http = 0;\n \n-\tstruct ref *refs = NULL;\n-\tstruct ref *ref = NULL;\n-\tstruct ref *last_ref = NULL;\n+\tif (last && !strcmp(service, last->service))\n+\t\treturn last;\n+\tfree_discovery(last);\n \n-\trefs_url = xmalloc(strlen(url) + 11);\n-\tsprintf(refs_url, \"%s/info/refs\", url);\n+\tstrbuf_addf(&buffer, \"%s/info/refs\", url);\n+\tif (!prefixcmp(url, \"http://\") || !prefixcmp(url, \"https://\")) {\n+\t\tis_http = 1;\n+\t\tif (!strchr(url, '?'))\n+\t\t\tstrbuf_addch(&buffer, '?');\n+\t\telse\n+\t\t\tstrbuf_addch(&buffer, '&');\n+\t\tstrbuf_addf(&buffer, \"service=%s\", service);\n+\t}\n+\trefs_url = strbuf_detach(&buffer, NULL);\n \n \tinit_walker();\n \thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n@@ -104,10 +130,86 @@ static struct ref *get_refs(void)\n \t\tdie(\"HTTP request failed\");\n \t}\n \n-\tdata = buffer.buf;\n+\tlast= xcalloc(1, sizeof(*last_discovery));\n+\tlast->service = service;\n+\tlast->buf_alloc = strbuf_detach(&buffer, &last->len);\n+\tlast->buf = last->buf_alloc;\n+\n+\tif (is_http && 5 <= last->len && last->buf[4] == '#') {\n+\t\t/* smart HTTP response; validate that the service\n+\t\t * pkt-line matches our request.\n+\t\t */\n+\t\tstruct strbuf exp = STRBUF_INIT;\n+\n+\t\tif (packet_get_line(&buffer, &last->buf, &last->len) <= 0)\n+\t\t\tdie(\"%s has invalid packet header\", refs_url);\n+\t\tif (buffer.len && buffer.buf[buffer.len - 1] == '\\n')\n+\t\t\tstrbuf_setlen(&buffer, buffer.len - 1);\n+\n+\t\tstrbuf_addf(&exp, \"# service=%s\", service);\n+\t\tif (strbuf_cmp(&exp, &buffer))\n+\t\t\tdie(\"invalid server response; got '%s'\", buffer.buf);\n+\t\tstrbuf_release(&exp);\n+\n+\t\t/* The header can include additional metadata lines, up\n+\t\t * until a packet flush marker. Ignore these now, but\n+\t\t * in the future we might start to scan them.\n+\t\t */\n+\t\tstrbuf_reset(&buffer);\n+\t\twhile (packet_get_line(&buffer, &last->buf, &last->len) > 0)\n+\t\t\tstrbuf_reset(&buffer);\n+\n+\t\tlast->proto_git = 1;\n+\t}\n+\n+\tfree(refs_url);\n+\tstrbuf_release(&buffer);\n+\tlast_discovery = last;\n+\treturn last;\n+}\n+\n+static int write_discovery(int fd, void *data)\n+{\n+\tstruct discovery *heads = data;\n+\tint err = 0;\n+\tif (write_in_full(fd, heads->buf, heads->len) != heads->len)\n+\t\terr = 1;\n+\tclose(fd);\n+\treturn err;\n+}\n+\n+static struct ref *parse_git_refs(struct discovery *heads)\n+{\n+\tstruct ref *list = NULL;\n+\tstruct async async;\n+\n+\tmemset(&async, 0, sizeof(async));\n+\tasync.proc = write_discovery;\n+\tasync.data = heads;\n+\n+\tif (start_async(&async))\n+\t\tdie(\"cannot start thread to parse advertised refs\");\n+\tget_remote_heads(async.out, &list, 0, NULL, 0, NULL);\n+\tclose(async.out);\n+\tif (finish_async(&async))\n+\t\tdie(\"ref parsing thread failed\");\n+\treturn list;\n+}\n+\n+static struct ref *parse_info_refs(struct discovery *heads)\n+{\n+\tchar *data, *start, *mid;\n+\tchar *ref_name;\n+\tint i = 0;\n+\n+\tstruct ref *refs = NULL;\n+\tstruct ref *ref = NULL;\n+\tstruct ref *last_ref = NULL;\n+\n+\tdata = heads->buf;\n \tstart = NULL;\n \tmid = data;\n-\twhile (i < buffer.len) {\n+\twhile (i < heads->len) {\n \t\tif (!start) {\n \t\t\tstart = &data[i];\n \t\t}\n@@ -131,8 +233,7 @@ static struct ref *get_refs(void)\n \t\ti++;\n \t}\n \n-\tstrbuf_release(&buffer);\n-\n+\tinit_walker();\n \tref = alloc_ref(\"HEAD\");\n \tif (!walker->fetch_ref(walker, ref) &&\n \t !resolve_remote_symref(ref, refs)) {\n@@ -142,11 +243,23 @@ static struct ref *get_refs(void)\n \t\tfree(ref);\n \t}\n \n-\tstrbuf_release(&buffer);\n-\tfree(refs_url);\n \treturn refs;\n }\n \n+static struct ref *get_refs(int for_push)\n+{\n+\tstruct discovery *heads;\n+\n+\tif (for_push)\n+\t\theads = discover_refs(\"git-receive-pack\");\n+\telse\n+\t\theads = discover_refs(\"git-upload-pack\");\n+\n+\tif (heads->proto_git)\n+\t\treturn parse_git_refs(heads);\n+\treturn parse_info_refs(heads);\n+}\n+\n static void output_refs(struct ref *refs)\n {\n \tstruct ref *posn;\n@@ -317,7 +430,8 @@ int main(int argc, const char **argv)\n \t\t\tparse_fetch(&buf);\n \n \t\t} else if (!strcmp(buf.buf, \"list\") || !prefixcmp(buf.buf, \"list \")) {\n-\t\t\toutput_refs(get_refs());\n+\t\t\tint for_push = !!strstr(buf.buf + 4, \"for-push\");\n+\t\t\toutput_refs(get_refs(for_push));\n \n \t\t} else if (!prefixcmp(buf.buf, \"push \")) {\n \t\t\tparse_push(&buf);\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126208","messageId":"1256774448-7625-23-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 22/26] Smart push over HTTP: client side","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:44Z","receivedAt":"2009-10-29T00:00:44Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The git-remote-curl backend detects if the remote server supports\nthe git-receive-pack service, and if so, runs git-send-pack in a\npipe to dump the command and pack data as a single POST request.\n\nThe advertisements from the server that were obtained during the\ndiscovery are passed into git-send-pack before the POST request\nstarts. This permits git-send-pack to operate largely unmodified.\n\nFor smaller packs (those under 1 MiB) a HTTP/1.0 POST with a\nContent-Length is used, permitting interaction with any server.\nThe 1 MiB limit is arbitrary, but is sufficent to fit most deltas\ncreated by human authors against text sources with the occasional\nsmall binary file (e.g. few KiB icon image). The configuration\noption http.postBuffer can be used to increase (or shink) this\nbuffer if the default is not sufficient.\n\nFor larger packs which cannot be spooled entirely into the helper's\nmemory space (due to http.postBuffer being too small), the POST\nrequest requires HTTP/1.1 and sets \"Transfer-Encoding: chunked\".\nThis permits the client to upload an unknown amount of data in one\nHTTP transaction without needing to pregenerate the entire pack\nfile locally.\n\nSigned-off-by: Shawn O. Pearce \nCC: Daniel Barkalow \n---\n Documentation/config.txt | 8 ++\n builtin-send-pack.c | 116 +++++++++++++++++++++-\n http.c | 13 ++-\n http.h | 2 +\n remote-curl.c | 235 +++++++++++++++++++++++++++++++++++++++++++++-\n send-pack.h | 3 +-\n sideband.c | 11 ++-\n transport.c | 1 +\n 8 files changed, 374 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex cd17814..7130d07 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1089,6 +1089,14 @@ http.maxRequests::\n \tHow many HTTP requests to launch in parallel. Can be overridden\n \tby the 'GIT_HTTP_MAX_REQUESTS' environment variable. Default is 5.\n \n+http.postBuffer::\n+\tMaximum size in bytes of the buffer used by smart HTTP\n+\ttransports when POSTing data to the remote system.\n+\tFor requests larger than this buffer size, HTTP/1.1 and\n+\tTransfer-Encoding: chunked is used to avoid creating a\n+\tmassive pack file locally. Default is 1 MiB, which is\n+\tsufficient for most requests.\n+\n http.lowSpeedLimit, http.lowSpeedTime::\n \tIf the HTTP transfer speed is less than 'http.lowSpeedLimit'\n \tfor longer than 'http.lowSpeedTime' seconds, the transfer is aborted.\ndiff --git a/builtin-send-pack.c b/builtin-send-pack.c\nindex 37e528e..a0fbad0 100644\n--- a/builtin-send-pack.c\n+++ b/builtin-send-pack.c\n@@ -2,9 +2,11 @@\n #include \"commit.h\"\n #include \"refs.h\"\n #include \"pkt-line.h\"\n+#include \"sideband.h\"\n #include \"run-command.h\"\n #include \"remote.h\"\n #include \"send-pack.h\"\n+#include \"quote.h\"\n \n static const char send_pack_usage[] =\n \"git send-pack [--all | --mirror] [--dry-run] [--force] [--receive-pack=] [--verbose] [--thin] [:] [...]\\n\"\n@@ -59,7 +61,7 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \tmemset(&po, 0, sizeof(po));\n \tpo.argv = argv;\n \tpo.in = -1;\n-\tpo.out = fd;\n+\tpo.out = args->stateless_rpc ? -1 : fd;\n \tpo.git_cmd = 1;\n \tif (start_command(&po))\n \t\tdie_errno(\"git pack-objects failed\");\n@@ -83,6 +85,20 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \t}\n \n \tclose(po.in);\n+\n+\tif (args->stateless_rpc) {\n+\t\tchar *buf = xmalloc(LARGE_PACKET_MAX);\n+\t\twhile (1) {\n+\t\t\tssize_t n = xread(po.out, buf, LARGE_PACKET_MAX);\n+\t\t\tif (n <= 0)\n+\t\t\t\tbreak;\n+\t\t\tsend_sideband(fd, -1, buf, n, LARGE_PACKET_MAX);\n+\t\t}\n+\t\tfree(buf);\n+\t\tclose(po.out);\n+\t\tpo.out = -1;\n+\t}\n+\n \tif (finish_command(&po))\n \t\treturn error(\"pack-objects died with strange error\");\n \treturn 0;\n@@ -303,6 +319,59 @@ static int refs_pushed(struct ref *ref)\n \treturn 0;\n }\n \n+static void print_helper_status(struct ref *ref)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tfor (; ref; ref = ref->next) {\n+\t\tconst char *msg = NULL;\n+\t\tconst char *res;\n+\n+\t\tswitch(ref->status) {\n+\t\tcase REF_STATUS_NONE:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"no match\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_OK:\n+\t\t\tres = \"ok\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_UPTODATE:\n+\t\t\tres = \"ok\";\n+\t\t\tmsg = \"up to date\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_REJECT_NONFASTFORWARD:\n+\t\t\tres = \"error\";\n+\t\t\tmsg = \"non-fast forward\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_REJECT_NODELETE:\n+\t\tcase REF_STATUS_REMOTE_REJECT:\n+\t\t\tres = \"error\";\n+\t\t\tbreak;\n+\n+\t\tcase REF_STATUS_EXPECTING_REPORT:\n+\t\tdefault:\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tstrbuf_reset(&buf);\n+\t\tstrbuf_addf(&buf, \"%s %s\", res, ref->name);\n+\t\tif (ref->remote_status)\n+\t\t\tmsg = ref->remote_status;\n+\t\tif (msg) {\n+\t\t\tstrbuf_addch(&buf, ' ');\n+\t\t\tquote_two_c_style(&buf, \"\", msg, 0);\n+\t\t}\n+\t\tstrbuf_addch(&buf, '\\n');\n+\n+\t\tsafe_write(1, buf.buf, buf.len);\n+\t}\n+\tstrbuf_release(&buf);\n+}\n+\n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n \t struct ref *remote_refs,\n@@ -310,6 +379,7 @@ int send_pack(struct send_pack_args *args,\n {\n \tint in = fd[0];\n \tint out = fd[1];\n+\tstruct strbuf req_buf = STRBUF_INIT;\n \tstruct ref *ref;\n \tint new_refs;\n \tint ask_for_status_report = 0;\n@@ -391,14 +461,14 @@ int send_pack(struct send_pack_args *args,\n \t\t\tchar *new_hex = sha1_to_hex(ref->new_sha1);\n \n \t\t\tif (ask_for_status_report) {\n-\t\t\t\tpacket_write(out, \"%s %s %s%c%s\",\n+\t\t\t\tpacket_buf_write(&req_buf, \"%s %s %s%c%s\",\n \t\t\t\t\told_hex, new_hex, ref->name, 0,\n \t\t\t\t\t\"report-status\");\n \t\t\t\task_for_status_report = 0;\n \t\t\t\texpect_status_report = 1;\n \t\t\t}\n \t\t\telse\n-\t\t\t\tpacket_write(out, \"%s %s %s\",\n+\t\t\t\tpacket_buf_write(&req_buf, \"%s %s %s\",\n \t\t\t\t\told_hex, new_hex, ref->name);\n \t\t}\n \t\tref->status = expect_status_report ?\n@@ -406,7 +476,17 @@ int send_pack(struct send_pack_args *args,\n \t\t\tREF_STATUS_OK;\n \t}\n \n-\tpacket_flush(out);\n+\tif (args->stateless_rpc) {\n+\t\tif (!args->dry_run) {\n+\t\t\tpacket_buf_flush(&req_buf);\n+\t\t\tsend_sideband(out, -1, req_buf.buf, req_buf.len, LARGE_PACKET_MAX);\n+\t\t}\n+\t} else {\n+\t\tsafe_write(out, req_buf.buf, req_buf.len);\n+\t\tpacket_flush(out);\n+\t}\n+\tstrbuf_release(&req_buf);\n+\n \tif (new_refs && !args->dry_run) {\n \t\tif (pack_objects(out, remote_refs, extra_have, args) < 0) {\n \t\t\tfor (ref = remote_refs; ref; ref = ref->next)\n@@ -414,11 +494,15 @@ int send_pack(struct send_pack_args *args,\n \t\t\treturn -1;\n \t\t}\n \t}\n+\tif (args->stateless_rpc && !args->dry_run)\n+\t\tpacket_flush(out);\n \n \tif (expect_status_report)\n \t\tret = receive_status(in, remote_refs);\n \telse\n \t\tret = 0;\n+\tif (args->stateless_rpc)\n+\t\tpacket_flush(out);\n \n \tif (ret < 0)\n \t\treturn ret;\n@@ -478,6 +562,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tstruct extra_have_objects extra_have;\n \tstruct ref *remote_refs, *local_refs;\n \tint ret;\n+\tint helper_status = 0;\n \tint send_all = 0;\n \tconst char *receivepack = \"git-receive-pack\";\n \tint flags;\n@@ -523,6 +608,14 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\targs.use_thin_pack = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(arg, \"--stateless-rpc\")) {\n+\t\t\t\targs.stateless_rpc = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!strcmp(arg, \"--helper-status\")) {\n+\t\t\t\thelper_status = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tusage(send_pack_usage);\n \t\t}\n \t\tif (!dest) {\n@@ -551,7 +644,14 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tconn = git_connect(fd, dest, receivepack, args.verbose ? CONNECT_VERBOSE : 0);\n+\tif (args.stateless_rpc) {\n+\t\tconn = NULL;\n+\t\tfd[0] = 0;\n+\t\tfd[1] = 1;\n+\t} else {\n+\t\tconn = git_connect(fd, dest, receivepack,\n+\t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n+\t}\n \n \tmemset(&extra_have, 0, sizeof(extra_have));\n \n@@ -575,12 +675,16 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \n \tret = send_pack(&args, fd, conn, remote_refs, &extra_have);\n \n+\tif (helper_status)\n+\t\tprint_helper_status(remote_refs);\n+\n \tclose(fd[1]);\n \tclose(fd[0]);\n \n \tret |= finish_connect(conn);\n \n-\tprint_push_status(dest, remote_refs);\n+\tif (!helper_status)\n+\t\tprint_push_status(dest, remote_refs);\n \n \tif (!args.dry_run && remote) {\n \t\tstruct ref *ref;\ndiff --git a/http.c b/http.c\nindex 23b2a19..ed6414a 100644\n--- a/http.c\n+++ b/http.c\n@@ -1,9 +1,11 @@\n #include \"http.h\"\n #include \"pack.h\"\n+#include \"sideband.h\"\n \n int data_received;\n int active_requests;\n int http_is_verbose;\n+size_t http_post_buffer = 16 * LARGE_PACKET_MAX;\n \n #ifdef USE_CURL_MULTI\n static int max_requests = -1;\n@@ -97,8 +99,6 @@ size_t fwrite_null(const void *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n \treturn eltsize * nmemb;\n }\n \n-static void finish_active_slot(struct active_request_slot *slot);\n-\n #ifdef USE_CURL_MULTI\n static void process_curl_messages(void)\n {\n@@ -174,6 +174,13 @@ static int http_options(const char *var, const char *value, void *cb)\n \tif (!strcmp(\"http.proxy\", var))\n \t\treturn git_config_string(&curl_http_proxy, var, value);\n \n+\tif (!strcmp(\"http.postbuffer\", var)) {\n+\t\thttp_post_buffer = git_config_int(var, value);\n+\t\tif (http_post_buffer < LARGE_PACKET_MAX)\n+\t\t\thttp_post_buffer = LARGE_PACKET_MAX;\n+\t\treturn 0;\n+\t}\n+\n \t/* Fall back on the default ones */\n \treturn git_default_config(var, value, cb);\n }\n@@ -638,7 +645,7 @@ void release_active_slot(struct active_request_slot *slot)\n #endif\n }\n \n-static void finish_active_slot(struct active_request_slot *slot)\n+void finish_active_slot(struct active_request_slot *slot)\n {\n \tclosedown_active_slot(slot);\n \tcurl_easy_getinfo(slot->curl, CURLINFO_HTTP_CODE, &slot->http_code);\ndiff --git a/http.h b/http.h\nindex 4c4e99c..f828e1d 100644\n--- a/http.h\n+++ b/http.h\n@@ -79,6 +79,7 @@ extern curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp);\n extern struct active_request_slot *get_active_slot(void);\n extern int start_active_slot(struct active_request_slot *slot);\n extern void run_active_slot(struct active_request_slot *slot);\n+extern void finish_active_slot(struct active_request_slot *slot);\n extern void finish_all_active_slots(void);\n extern void release_active_slot(struct active_request_slot *slot);\n \n@@ -94,6 +95,7 @@ extern void http_cleanup(void);\n extern int data_received;\n extern int active_requests;\n extern int http_is_verbose;\n+extern size_t http_post_buffer;\n \n extern char curl_errorstr[CURL_ERROR_SIZE];\n \ndiff --git a/remote-curl.c b/remote-curl.c\nindex 3917d45..f1206cb 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -6,6 +6,7 @@\n #include \"exec_cmd.h\"\n #include \"run-command.h\"\n #include \"pkt-line.h\"\n+#include \"sideband.h\"\n \n static struct remote *remote;\n static const char *url;\n@@ -16,7 +17,8 @@ struct options {\n \tunsigned long depth;\n \tunsigned progress : 1,\n \t\tfollowtags : 1,\n-\t\tdry_run : 1;\n+\t\tdry_run : 1,\n+\t\tthin : 1;\n };\n static struct options options;\n \n@@ -274,6 +276,188 @@ static void output_refs(struct ref *refs)\n \tfree_refs(refs);\n }\n \n+struct rpc_state {\n+\tconst char *service_name;\n+\tconst char **argv;\n+\tchar *service_url;\n+\tchar *hdr_content_type;\n+\tchar *hdr_accept;\n+\tchar *buf;\n+\tsize_t alloc;\n+\tsize_t len;\n+\tsize_t pos;\n+\tint in;\n+\tint out;\n+\tstruct strbuf result;\n+};\n+\n+static size_t rpc_out(void *ptr, size_t eltsize,\n+\t\tsize_t nmemb, void *buffer_)\n+{\n+\tsize_t max = eltsize * nmemb;\n+\tstruct rpc_state *rpc = buffer_;\n+\tsize_t avail = rpc->len - rpc->pos;\n+\n+\tif (!avail) {\n+\t\tavail = packet_read_line(rpc->out, rpc->buf, rpc->alloc);\n+\t\tif (!avail)\n+\t\t\treturn 0;\n+\t\trpc->pos = 0;\n+\t\trpc->len = avail;\n+\t}\n+\n+\tif (max < avail);\n+\t\tavail = max;\n+\tmemcpy(ptr, rpc->buf + rpc->pos, avail);\n+\trpc->pos += avail;\n+\treturn avail;\n+}\n+\n+static size_t rpc_in(const void *ptr, size_t eltsize,\n+\t\tsize_t nmemb, void *buffer_)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct rpc_state *rpc = buffer_;\n+\twrite_or_die(rpc->in, ptr, size);\n+\treturn size;\n+}\n+\n+static int post_rpc(struct rpc_state *rpc)\n+{\n+\tstruct active_request_slot *slot;\n+\tstruct slot_results results;\n+\tstruct curl_slist *headers = NULL;\n+\tint err = 0, large_request = 0;\n+\n+\t/* Try to load the entire request, if we can fit it into the\n+\t * allocated buffer space we can use HTTP/1.0 and avoid the\n+\t * chunked encoding mess.\n+\t */\n+\twhile (1) {\n+\t\tsize_t left = rpc->alloc - rpc->len;\n+\t\tchar *buf = rpc->buf + rpc->len;\n+\t\tint n;\n+\n+\t\tif (left < LARGE_PACKET_MAX) {\n+\t\t\tlarge_request = 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tn = packet_read_line(rpc->out, buf, left);\n+\t\tif (!n)\n+\t\t\tbreak;\n+\t\trpc->len += n;\n+\t}\n+\n+\tslot = get_active_slot();\n+\tslot->results = &results;\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, rpc->service_url);\n+\n+\theaders = curl_slist_append(headers, rpc->hdr_content_type);\n+\theaders = curl_slist_append(headers, rpc->hdr_accept);\n+\n+\tif (large_request) {\n+\t\t/* The request body is large and the size cannot be predicted.\n+\t\t * We must use chunked encoding to send it.\n+\t\t */\n+\t\theaders = curl_slist_append(headers, \"Expect: 100-continue\");\n+\t\theaders = curl_slist_append(headers, \"Transfer-Encoding: chunked\");\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc);\n+\t\tif (options.verbosity > 1) {\n+\t\t\tfprintf(stderr, \"POST %s (chunked)\\n\", rpc->service_name);\n+\t\t\tfflush(stderr);\n+\t\t}\n+\n+\t} else {\n+\t\t/* We know the complete request size in advance, use the\n+\t\t * more normal Content-Length approach.\n+\t\t */\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, rpc->buf);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, rpc->len);\n+\t\tif (options.verbosity > 1) {\n+\t\t\tfprintf(stderr, \"POST %s (%lu bytes)\\n\",\n+\t\t\t\trpc->service_name, (unsigned long)rpc->len);\n+\t\t\tfflush(stderr);\n+\t\t}\n+\t}\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, rpc);\n+\n+\tslot->curl_result = curl_easy_perform(slot->curl);\n+\tfinish_active_slot(slot);\n+\n+\tif (results.curl_result != CURLE_OK) {\n+\t\terr |= error(\"RPC failed; result=%d, HTTP code = %ld\",\n+\t\t\tresults.curl_result, results.http_code);\n+\t}\n+\n+\tcurl_slist_free_all(headers);\n+\treturn err;\n+}\n+\n+static int rpc_service(struct rpc_state *rpc, struct discovery *heads)\n+{\n+\tconst char *svc = rpc->service_name;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct child_process client;\n+\tint err = 0;\n+\n+\tinit_walker();\n+\tmemset(&client, 0, sizeof(client));\n+\tclient.in = -1;\n+\tclient.out = -1;\n+\tclient.git_cmd = 1;\n+\tclient.argv = rpc->argv;\n+\tif (start_command(&client))\n+\t\texit(1);\n+\tif (heads)\n+\t\twrite_or_die(client.in, heads->buf, heads->len);\n+\n+\trpc->alloc = http_post_buffer;\n+\trpc->buf = xmalloc(rpc->alloc);\n+\trpc->in = client.in;\n+\trpc->out = client.out;\n+\tstrbuf_init(&rpc->result, 0);\n+\n+\tstrbuf_addf(&buf, \"%s/%s\", url, svc);\n+\trpc->service_url = strbuf_detach(&buf, NULL);\n+\n+\tstrbuf_addf(&buf, \"Content-Type: application/x-%s-request\", svc);\n+\trpc->hdr_content_type = strbuf_detach(&buf, NULL);\n+\n+\tstrbuf_addf(&buf, \"Accept: application/x-%s-response\", svc);\n+\trpc->hdr_accept = strbuf_detach(&buf, NULL);\n+\n+\twhile (!err) {\n+\t\tint n = packet_read_line(rpc->out, rpc->buf, rpc->alloc);\n+\t\tif (!n)\n+\t\t\tbreak;\n+\t\trpc->pos = 0;\n+\t\trpc->len = n;\n+\t\terr |= post_rpc(rpc);\n+\t}\n+\tstrbuf_read(&rpc->result, client.out, 0);\n+\n+\tclose(client.in);\n+\tclose(client.out);\n+\tclient.in = -1;\n+\tclient.out = -1;\n+\n+\terr |= finish_command(&client);\n+\tfree(rpc->service_url);\n+\tfree(rpc->hdr_content_type);\n+\tfree(rpc->hdr_accept);\n+\tfree(rpc->buf);\n+\tstrbuf_release(&buf);\n+\treturn err;\n+}\n+\n static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n {\n \tchar **targets = xmalloc(nr_heads * sizeof(char*));\n@@ -371,6 +555,52 @@ static int push_dav(int nr_spec, char **specs)\n \treturn 0;\n }\n \n+static int push_git(struct discovery *heads, int nr_spec, char **specs)\n+{\n+\tstruct rpc_state rpc;\n+\tconst char **argv;\n+\tint argc = 0, i, err;\n+\n+\targv = xmalloc((10 + nr_spec) * sizeof(char*));\n+\targv[argc++] = \"send-pack\";\n+\targv[argc++] = \"--stateless-rpc\";\n+\targv[argc++] = \"--helper-status\";\n+\tif (options.thin)\n+\t\targv[argc++] = \"--thin\";\n+\tif (options.dry_run)\n+\t\targv[argc++] = \"--dry-run\";\n+\tif (options.verbosity > 1)\n+\t\targv[argc++] = \"--verbose\";\n+\targv[argc++] = url;\n+\tfor (i = 0; i < nr_spec; i++)\n+\t\targv[argc++] = specs[i];\n+\targv[argc++] = NULL;\n+\n+\tmemset(&rpc, 0, sizeof(rpc));\n+\trpc.service_name = \"git-receive-pack\",\n+\trpc.argv = argv;\n+\n+\terr = rpc_service(&rpc, heads);\n+\tif (rpc.result.len)\n+\t\tsafe_write(1, rpc.result.buf, rpc.result.len);\n+\tstrbuf_release(&rpc.result);\n+\tfree(argv);\n+\treturn err;\n+}\n+\n+static int push(int nr_spec, char **specs)\n+{\n+\tstruct discovery *heads = discover_refs(\"git-receive-pack\");\n+\tint ret;\n+\n+\tif (heads->proto_git)\n+\t\tret = push_git(heads, nr_spec, specs);\n+\telse\n+\t\tret = push_dav(nr_spec, specs);\n+\tfree_discovery(heads);\n+\treturn ret;\n+}\n+\n static void parse_push(struct strbuf *buf)\n {\n \tchar **specs = NULL;\n@@ -391,7 +621,7 @@ static void parse_push(struct strbuf *buf)\n \t\t\tbreak;\n \t} while (1);\n \n-\tif (push_dav(nr_spec, specs))\n+\tif (push(nr_spec, specs))\n \t\texit(128); /* error already reported */\n \tfor (i = 0; i < nr_spec; i++)\n \t\tfree(specs[i]);\n@@ -414,6 +644,7 @@ int main(int argc, const char **argv)\n \n \toptions.verbosity = 1;\n \toptions.progress = !!isatty(2);\n+\toptions.thin = 1;\n \n \tremote = remote_get(argv[1]);\n \ndiff --git a/send-pack.h b/send-pack.h\nindex 8b3cf02..28141ac 100644\n--- a/send-pack.h\n+++ b/send-pack.h\n@@ -8,7 +8,8 @@ struct send_pack_args {\n \t\tforce_update:1,\n \t\tuse_thin_pack:1,\n \t\tuse_ofs_delta:1,\n-\t\tdry_run:1;\n+\t\tdry_run:1,\n+\t\tstateless_rpc:1;\n };\n \n int send_pack(struct send_pack_args *args,\ndiff --git a/sideband.c b/sideband.c\nindex 899b1ff..d5ffa1c 100644\n--- a/sideband.c\n+++ b/sideband.c\n@@ -135,9 +135,14 @@ ssize_t send_sideband(int fd, int band, const char *data, ssize_t sz, int packet\n \t\tn = sz;\n \t\tif (packet_max - 5 < n)\n \t\t\tn = packet_max - 5;\n-\t\tsprintf(hdr, \"%04x\", n + 5);\n-\t\thdr[4] = band;\n-\t\tsafe_write(fd, hdr, 5);\n+\t\tif (0 <= band) {\n+\t\t\tsprintf(hdr, \"%04x\", n + 5);\n+\t\t\thdr[4] = band;\n+\t\t\tsafe_write(fd, hdr, 5);\n+\t\t} else {\n+\t\t\tsprintf(hdr, \"%04x\", n + 4);\n+\t\t\tsafe_write(fd, hdr, 4);\n+\t\t}\n \t\tsafe_write(fd, p, n);\n \t\tp += n;\n \t\tsz -= n;\ndiff --git a/transport.c b/transport.c\nindex 6d9652d..2ff1650 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -731,6 +731,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\t\t\t NULL);\n \t}\n \n+\tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n \targs.force_update = !!(flags & TRANSPORT_PUSH_FORCE);\n \targs.use_thin_pack = data->thin;\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126211","messageId":"1256774448-7625-24-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 23/26] Smart fetch over HTTP: client side","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:45Z","receivedAt":"2009-10-29T00:00:45Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The git-remote-curl backend detects if the remote server supports\nthe git-upload-pack service, and if so, runs git-fetch-pack locally\nin a pipe to generate the want/have commands.\n\nThe advertisements from the server that were obtained during the\ndiscovery are passed into git-fetch-pack before the POST request\nstarts, permitting server capability discovery and enablement.\n\nCommon objects that are discovered are appended onto the request as\nhave lines and are sent again on the next request. This allows the\nremote side to reinitialize its in-memory list of common objects\nduring the next request.\n\nBecause all requests are relatively short, below git-remote-curl's\n1 MiB buffer limit, requests will use the standard Content-Length\nheader and be valid HTTP/1.0 POST requests. This makes the fetch\nclient more tolerant of proxy servers which don't support HTTP/1.1\nor the chunked transfer encoding.\n\nSigned-off-by: Shawn O. Pearce \nCC: Daniel Barkalow \n---\n builtin-fetch-pack.c | 110 ++++++++++++++++++++++++++++++++++++++++++--------\n fetch-pack.h | 3 +-\n remote-curl.c | 69 +++++++++++++++++++++++++++++--\n 3 files changed, 160 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex 615f549..8ed4a6f 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -165,6 +165,24 @@ enum ack_type {\n \tACK_ready\n };\n \n+static void consume_shallow_list(int fd)\n+{\n+\tif (args.stateless_rpc && args.depth > 0) {\n+\t\t/* If we sent a depth we will get back \"duplicate\"\n+\t\t * shallow and unshallow commands every time there\n+\t\t * is a block of have lines exchanged.\n+\t\t */\n+\t\tchar line[1000];\n+\t\twhile (packet_read_line(fd, line, sizeof(line))) {\n+\t\t\tif (!prefixcmp(line, \"shallow \"))\n+\t\t\t\tcontinue;\n+\t\t\tif (!prefixcmp(line, \"unshallow \"))\n+\t\t\t\tcontinue;\n+\t\t\tdie(\"git fetch-pack: expected shallow list\");\n+\t\t}\n+\t}\n+}\n+\n static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n {\n \tstatic char line[1000];\n@@ -190,6 +208,15 @@ static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n \tdie(\"git fetch_pack: expected ACK/NAK, got '%s'\", line);\n }\n \n+static void send_request(int fd, struct strbuf *buf)\n+{\n+\tif (args.stateless_rpc) {\n+\t\tsend_sideband(fd, -1, buf->buf, buf->len, LARGE_PACKET_MAX);\n+\t\tpacket_flush(fd);\n+\t} else\n+\t\tsafe_write(fd, buf->buf, buf->len);\n+}\n+\n static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t struct ref *refs)\n {\n@@ -199,7 +226,10 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tunsigned in_vain = 0;\n \tint got_continue = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n+\tsize_t state_len = 0;\n \n+\tif (args.stateless_rpc && multi_ack == 1)\n+\t\tdie(\"--stateless-rpc requires multi_ack_detailed\");\n \tif (marked)\n \t\tfor_each_ref(clear_marks, NULL);\n \tmarked = 1;\n@@ -256,13 +286,13 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tif (args.depth > 0)\n \t\tpacket_buf_write(&req_buf, \"deepen %d\", args.depth);\n \tpacket_buf_flush(&req_buf);\n-\n-\tsafe_write(fd[1], req_buf.buf, req_buf.len);\n+\tstate_len = req_buf.len;\n \n \tif (args.depth > 0) {\n \t\tchar line[1024];\n \t\tunsigned char sha1[20];\n \n+\t\tsend_request(fd[1], &req_buf);\n \t\twhile (packet_read_line(fd[0], line, sizeof(line))) {\n \t\t\tif (!prefixcmp(line, \"shallow \")) {\n \t\t\t\tif (get_sha1_hex(line + 8, sha1))\n@@ -284,28 +314,40 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t\t}\n \t\t\tdie(\"expected shallow/unshallow, got %s\", line);\n \t\t}\n+\t} else if (!args.stateless_rpc)\n+\t\tsend_request(fd[1], &req_buf);\n+\n+\tif (!args.stateless_rpc) {\n+\t\t/* If we aren't using the stateless-rpc interface\n+\t\t * we don't need to retain the headers.\n+\t\t */\n+\t\tstrbuf_setlen(&req_buf, 0);\n+\t\tstate_len = 0;\n \t}\n \n \tflushes = 0;\n \tretval = -1;\n \twhile ((sha1 = get_rev())) {\n-\t\tpacket_write(fd[1], \"have %s\\n\", sha1_to_hex(sha1));\n+\t\tpacket_buf_write(&req_buf, \"have %s\\n\", sha1_to_hex(sha1));\n \t\tif (args.verbose)\n \t\t\tfprintf(stderr, \"have %s\\n\", sha1_to_hex(sha1));\n \t\tin_vain++;\n \t\tif (!(31 & ++count)) {\n \t\t\tint ack;\n \n-\t\t\tpacket_flush(fd[1]);\n+\t\t\tpacket_buf_flush(&req_buf);\n+\t\t\tsend_request(fd[1], &req_buf);\n+\t\t\tstrbuf_setlen(&req_buf, state_len);\n \t\t\tflushes++;\n \n \t\t\t/*\n \t\t\t * We keep one window \"ahead\" of the other side, and\n \t\t\t * will wait for an ACK only on the next one\n \t\t\t */\n-\t\t\tif (count == 32)\n+\t\t\tif (!args.stateless_rpc && count == 32)\n \t\t\t\tcontinue;\n \n+\t\t\tconsume_shallow_list(fd[0]);\n \t\t\tdo {\n \t\t\t\tack = get_ack(fd[0], result_sha1);\n \t\t\t\tif (args.verbose && ack)\n@@ -322,6 +364,17 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t\t\tcase ACK_continue: {\n \t\t\t\t\tstruct commit *commit =\n \t\t\t\t\t\tlookup_commit(result_sha1);\n+\t\t\t\t\tif (args.stateless_rpc\n+\t\t\t\t\t && ack == ACK_common\n+\t\t\t\t\t && !(commit->object.flags & COMMON)) {\n+\t\t\t\t\t\t/* We need to replay the have for this object\n+\t\t\t\t\t\t * on the next RPC request so the peer knows\n+\t\t\t\t\t\t * it is in common with us.\n+\t\t\t\t\t\t */\n+\t\t\t\t\t\tconst char *hex = sha1_to_hex(result_sha1);\n+\t\t\t\t\t\tpacket_buf_write(&req_buf, \"have %s\\n\", hex);\n+\t\t\t\t\t\tstate_len = req_buf.len;\n+\t\t\t\t\t}\n \t\t\t\t\tmark_common(commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tin_vain = 0;\n@@ -339,7 +392,8 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t}\n \t}\n done:\n-\tpacket_write(fd[1], \"done\\n\");\n+\tpacket_buf_write(&req_buf, \"done\\n\");\n+\tsend_request(fd[1], &req_buf);\n \tif (args.verbose)\n \t\tfprintf(stderr, \"done\\n\");\n \tif (retval != 0) {\n@@ -348,6 +402,7 @@ done:\n \t}\n \tstrbuf_release(&req_buf);\n \n+\tconsume_shallow_list(fd[0]);\n \twhile (flushes || multi_ack) {\n \t\tint ack = get_ack(fd[0], result_sha1);\n \t\tif (ack) {\n@@ -672,6 +727,8 @@ static struct ref *do_fetch_pack(int fd[2],\n \t\t\t */\n \t\t\twarning(\"no common commits\");\n \n+\tif (args.stateless_rpc)\n+\t\tpacket_flush(fd[1]);\n \tif (get_pack(fd, pack_lockfile))\n \t\tdie(\"git fetch-pack: fetch failed.\");\n \n@@ -742,6 +799,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tstruct ref *ref = NULL;\n \tchar *dest = NULL, **heads;\n \tint fd[2];\n+\tchar *pack_lockfile = NULL;\n+\tchar **pack_lockfile_ptr = NULL;\n \tstruct child_process *conn;\n \n \tnr_heads = 0;\n@@ -791,6 +850,15 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\targs.no_progress = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(\"--stateless-rpc\", arg)) {\n+\t\t\t\targs.stateless_rpc = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!strcmp(\"--lock-pack\", arg)) {\n+\t\t\t\targs.lock_pack = 1;\n+\t\t\t\tpack_lockfile_ptr = &pack_lockfile;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tusage(fetch_pack_usage);\n \t\t}\n \t\tdest = (char *)arg;\n@@ -801,19 +869,27 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tif (!dest)\n \t\tusage(fetch_pack_usage);\n \n-\tconn = git_connect(fd, (char *)dest, args.uploadpack,\n-\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n-\tif (conn) {\n-\t\tget_remote_heads(fd[0], &ref, 0, NULL, 0, NULL);\n-\n-\t\tref = fetch_pack(&args, fd, conn, ref, dest, nr_heads, heads, NULL);\n-\t\tclose(fd[0]);\n-\t\tclose(fd[1]);\n-\t\tif (finish_connect(conn))\n-\t\t\tref = NULL;\n+\tif (args.stateless_rpc) {\n+\t\tconn = NULL;\n+\t\tfd[0] = 0;\n+\t\tfd[1] = 1;\n \t} else {\n-\t\tref = NULL;\n+\t\tconn = git_connect(fd, (char *)dest, args.uploadpack,\n+\t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n+\t}\n+\n+\tget_remote_heads(fd[0], &ref, 0, NULL, 0, NULL);\n+\n+\tref = fetch_pack(&args, fd, conn, ref, dest,\n+\t\tnr_heads, heads, pack_lockfile_ptr);\n+\tif (pack_lockfile) {\n+\t\tprintf(\"lock %s\\n\", pack_lockfile);\n+\t\tfflush(stdout);\n \t}\n+\tclose(fd[0]);\n+\tclose(fd[1]);\n+\tif (finish_connect(conn))\n+\t\tref = NULL;\n \tret = !ref;\n \n \tif (!ret && nr_heads) {\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 8bd9c32..fbe85ac 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -13,7 +13,8 @@ struct fetch_pack_args\n \t\tfetch_all:1,\n \t\tverbose:1,\n \t\tno_progress:1,\n-\t\tinclude_tag:1;\n+\t\tinclude_tag:1,\n+\t\tstateless_rpc:1;\n };\n \n struct ref *fetch_pack(struct fetch_pack_args *args,\ndiff --git a/remote-curl.c b/remote-curl.c\nindex f1206cb..0eb6fc4 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -45,7 +45,7 @@ static int set_option(const char *name, const char *value)\n \t\t\toptions.progress = 0;\n \t\telse\n \t\t\treturn -1;\n-\t\treturn 1 /* TODO implement later */;\n+\t\treturn 0;\n \t}\n \telse if (!strcmp(name, \"depth\")) {\n \t\tchar *end;\n@@ -53,7 +53,7 @@ static int set_option(const char *name, const char *value)\n \t\tif (value == end || *end)\n \t\t\treturn -1;\n \t\toptions.depth = v;\n-\t\treturn 1 /* TODO implement later */;\n+\t\treturn 0;\n \t}\n \telse if (!strcmp(name, \"followtags\")) {\n \t\tif (!strcmp(value, \"true\"))\n@@ -62,7 +62,7 @@ static int set_option(const char *name, const char *value)\n \t\t\toptions.followtags = 0;\n \t\telse\n \t\t\treturn -1;\n-\t\treturn 1 /* TODO implement later */;\n+\t\treturn 0;\n \t}\n \telse if (!strcmp(name, \"dry-run\")) {\n \t\tif (!strcmp(value, \"true\"))\n@@ -463,6 +463,8 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n \tchar **targets = xmalloc(nr_heads * sizeof(char*));\n \tint ret, i;\n \n+\tif (options.depth)\n+\t\tdie(\"dumb http transport does not support --depth\");\n \tfor (i = 0; i < nr_heads; i++)\n \t\ttargets[i] = xstrdup(sha1_to_hex(to_fetch[i]->old_sha1));\n \n@@ -481,6 +483,65 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n \treturn ret ? error(\"Fetch failed.\") : 0;\n }\n \n+static int fetch_git(struct discovery *heads,\n+\tint nr_heads, struct ref **to_fetch)\n+{\n+\tstruct rpc_state rpc;\n+\tchar *depth_arg = NULL;\n+\tconst char **argv;\n+\tint argc = 0, i, err;\n+\n+\targv = xmalloc((15 + nr_heads) * sizeof(char*));\n+\targv[argc++] = \"fetch-pack\";\n+\targv[argc++] = \"--stateless-rpc\";\n+\targv[argc++] = \"--lock-pack\";\n+\tif (options.followtags)\n+\t\targv[argc++] = \"--include-tag\";\n+\tif (options.thin)\n+\t\targv[argc++] = \"--thin\";\n+\tif (options.verbosity >= 3) {\n+\t\targv[argc++] = \"-v\";\n+\t\targv[argc++] = \"-v\";\n+\t}\n+\tif (!options.progress)\n+\t\targv[argc++] = \"--no-progress\";\n+\tif (options.depth) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tstrbuf_addf(&buf, \"--depth=%lu\", options.depth);\n+\t\tdepth_arg = strbuf_detach(&buf, NULL);\n+\t\targv[argc++] = depth_arg;\n+\t}\n+\targv[argc++] = url;\n+\tfor (i = 0; i < nr_heads; i++) {\n+\t\tstruct ref *ref = to_fetch[i];\n+\t\tif (!ref->name || !*ref->name)\n+\t\t\tdie(\"cannot fetch by sha1 over smart http\");\n+\t\targv[argc++] = ref->name;\n+\t}\n+\targv[argc++] = NULL;\n+\n+\tmemset(&rpc, 0, sizeof(rpc));\n+\trpc.service_name = \"git-upload-pack\",\n+\trpc.argv = argv;\n+\n+\terr = rpc_service(&rpc, heads);\n+\tif (rpc.result.len)\n+\t\tsafe_write(1, rpc.result.buf, rpc.result.len);\n+\tstrbuf_release(&rpc.result);\n+\tfree(argv);\n+\tfree(depth_arg);\n+\treturn err;\n+}\n+\n+static int fetch(int nr_heads, struct ref **to_fetch)\n+{\n+\tstruct discovery *d = discover_refs(\"git-upload-pack\");\n+\tif (d->proto_git)\n+\t\treturn fetch_git(d, nr_heads, to_fetch);\n+\telse\n+\t\treturn fetch_dumb(nr_heads, to_fetch);\n+}\n+\n static void parse_fetch(struct strbuf *buf)\n {\n \tstruct ref **to_fetch = NULL;\n@@ -523,7 +584,7 @@ static void parse_fetch(struct strbuf *buf)\n \t\t\tbreak;\n \t} while (1);\n \n-\tif (fetch_dumb(nr_heads, to_fetch))\n+\tif (fetch(nr_heads, to_fetch))\n \t\texit(128); /* error already reported */\n \tfree_refs(list_head);\n \tfree(to_fetch);\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126207","messageId":"1256774448-7625-25-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 24/26] Smart HTTP fetch: gzip requests","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:46Z","receivedAt":"2009-10-29T00:00:46Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The upload-pack requests are mostly plain text and they compress\nrather well. Deflating them with Content-Encoding: gzip can easily\ndrop the size of the request by 50%, reducing the amount of data\nto transfer as we negotiate the common commits.\n\nSigned-off-by: Shawn O. Pearce \nCC: Daniel Barkalow \n---\n remote-curl.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 50 insertions(+), 0 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 0eb6fc4..0d7cf16 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -289,6 +289,7 @@ struct rpc_state {\n \tint in;\n \tint out;\n \tstruct strbuf result;\n+\tunsigned gzip_request : 1;\n };\n \n static size_t rpc_out(void *ptr, size_t eltsize,\n@@ -327,6 +328,8 @@ static int post_rpc(struct rpc_state *rpc)\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n \tstruct curl_slist *headers = NULL;\n+\tint use_gzip = rpc->gzip_request;\n+\tchar *gzip_body = NULL;\n \tint err = 0, large_request = 0;\n \n \t/* Try to load the entire request, if we can fit it into the\n@@ -340,6 +343,7 @@ static int post_rpc(struct rpc_state *rpc)\n \n \t\tif (left < LARGE_PACKET_MAX) {\n \t\t\tlarge_request = 1;\n+\t\t\tuse_gzip = 0;\n \t\t\tbreak;\n \t\t}\n \n@@ -355,6 +359,7 @@ static int post_rpc(struct rpc_state *rpc)\n \tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n \tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n \tcurl_easy_setopt(slot->curl, CURLOPT_URL, rpc->service_url);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, \"\");\n \n \theaders = curl_slist_append(headers, rpc->hdr_content_type);\n \theaders = curl_slist_append(headers, rpc->hdr_accept);\n@@ -372,6 +377,49 @@ static int post_rpc(struct rpc_state *rpc)\n \t\t\tfflush(stderr);\n \t\t}\n \n+\t} else if (use_gzip && 1024 < rpc->len) {\n+\t\t/* The client backend isn't giving us compressed data so\n+\t\t * we can try to deflate it ourselves, this may save on.\n+\t\t * the transfer time.\n+\t\t */\n+\t\tsize_t size;\n+\t\tz_stream stream;\n+\t\tint ret;\n+\n+\t\tmemset(&stream, 0, sizeof(stream));\n+\t\tret = deflateInit2(&stream, Z_BEST_COMPRESSION,\n+\t\t\t\tZ_DEFLATED, (15 + 16),\n+\t\t\t\t8, Z_DEFAULT_STRATEGY);\n+\t\tif (ret != Z_OK)\n+\t\t\tdie(\"cannot deflate request; zlib init error %d\", ret);\n+\t\tsize = deflateBound(&stream, rpc->len);\n+\t\tgzip_body = xmalloc(size);\n+\n+\t\tstream.next_in = (unsigned char *)rpc->buf;\n+\t\tstream.avail_in = rpc->len;\n+\t\tstream.next_out = (unsigned char *)gzip_body;\n+\t\tstream.avail_out = size;\n+\n+\t\tret = deflate(&stream, Z_FINISH);\n+\t\tif (ret != Z_STREAM_END)\n+\t\t\tdie(\"cannot deflate request; zlib deflate error %d\", ret);\n+\n+\t\tret = deflateEnd(&stream);\n+\t\tif (ret != Z_OK)\n+\t\t\tdie(\"cannot deflate request; zlib end error %d\", ret);\n+\n+\t\tsize = stream.total_out;\n+\n+\t\theaders = curl_slist_append(headers, \"Content-Encoding: gzip\");\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, size);\n+\n+\t\tif (options.verbosity > 1) {\n+\t\t\tfprintf(stderr, \"POST %s (gzip %lu to %lu bytes)\\n\",\n+\t\t\t\trpc->service_name,\n+\t\t\t\t(unsigned long)rpc->len, (unsigned long)size);\n+\t\t\tfflush(stderr);\n+\t\t}\n \t} else {\n \t\t/* We know the complete request size in advance, use the\n \t\t * more normal Content-Length approach.\n@@ -398,6 +446,7 @@ static int post_rpc(struct rpc_state *rpc)\n \t}\n \n \tcurl_slist_free_all(headers);\n+\tfree(gzip_body);\n \treturn err;\n }\n \n@@ -523,6 +572,7 @@ static int fetch_git(struct discovery *heads,\n \tmemset(&rpc, 0, sizeof(rpc));\n \trpc.service_name = \"git-upload-pack\",\n \trpc.argv = argv;\n+\trpc.gzip_request = 1;\n \n \terr = rpc_service(&rpc, heads);\n \tif (rpc.result.len)\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126209","messageId":"1256774448-7625-26-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 25/26] t5540-http-push: remove redundant fetches","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:47Z","receivedAt":"2009-10-29T00:00:47Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"From: Tay Ray Chuan \n\nSigned-off-by: Tay Ray Chuan \nSigned-off-by: Shawn O. Pearce \n---\n t/t5540-http-push.sh | 2 --\n 1 files changed, 0 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5540-http-push.sh b/t/t5540-http-push.sh\nindex 2ece661..28a746e 100755\n--- a/t/t5540-http-push.sh\n+++ b/t/t5540-http-push.sh\n@@ -116,9 +116,7 @@ test_expect_success 'create and delete remote branch' '\n \ttest_tick &&\n \tgit commit -m dev &&\n \tgit push origin dev &&\n-\tgit fetch &&\n \tgit push origin :dev &&\n-\tgit fetch &&\n \ttest_must_fail git show-ref --verify refs/remotes/origin/dev\n '\n \n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126203","messageId":"1256774448-7625-27-git-send-email-spearce@spearce.org","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"[RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T00:00:48Z","receivedAt":"2009-10-29T00:00:48Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The top level directory \"/git/\" of the test Apache server is mapped\nthrough our git-http-backend CGI, but uses the same underlying\nrepository space as the server's document root. This is the most\nsimple installation possible.\n\nServer logs are checked to verify the client has accessed only the\nsmart URLs during the test. During fetch testing the headers are\nalso logged from libcurl to ensure we are making a reasonably sane\nHTTP request, and getting back reasonably sane response headers\nfrom the CGI.\n\nSigned-off-by: Shawn O. Pearce \n---\n t/lib-httpd/apache.conf | 20 +++++++++\n t/t5540-http-push.sh | 2 +-\n t/t5541-http-push.sh | 103 +++++++++++++++++++++++++++++++++++++++++++++++\n t/t5550-http-fetch.sh | 8 +++-\n t/t5551-http-fetch.sh | 87 +++++++++++++++++++++++++++++++++++++++\n 5 files changed, 218 insertions(+), 2 deletions(-)\n create mode 100755 t/t5541-http-push.sh\n create mode 100755 t/t5551-http-fetch.sh\n\ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 21aa42f..2098ce0 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -8,6 +8,26 @@ ErrorLog error.log\n \n \tLoadModule log_config_module modules/mod_log_config.so\n \n+\n+\tLoadModule cgi_module modules/mod_cgi.so\n+\n+\n+\tLoadModule alias_module modules/mod_alias.so\n+\n+\n+\tLoadModule env_module modules/mod_env.so\n+\n+\n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\n+ScriptAlias /git/ ${GIT_EXEC_PATH}/git-http-backend/\n+\n+\tOptions None\n+\n+\n+\tOptions ExecCGI\n+\n \n \n LoadModule ssl_module modules/mod_ssl.so\ndiff --git a/t/t5540-http-push.sh b/t/t5540-http-push.sh\nindex 28a746e..5c0f4d7 100755\n--- a/t/t5540-http-push.sh\n+++ b/t/t5540-http-push.sh\n@@ -3,7 +3,7 @@\n # Copyright (c) 2008 Clemens Buchacher \n #\n \n-test_description='test http-push\n+test_description='test WebDAV http-push\n \n This test runs various sanity checks on http-push.'\n \ndiff --git a/t/t5541-http-push.sh b/t/t5541-http-push.sh\nnew file mode 100755\nindex 0000000..690c466\n--- /dev/null\n+++ b/t/t5541-http-push.sh\n@@ -0,0 +1,103 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2008 Clemens Buchacher \n+#\n+\n+test_description='test smart pushing over http via http-backend'\n+. ./test-lib.sh\n+\n+if test -n \"$NO_CURL\"; then\n+\tsay 'skipping test, git built without http support'\n+\ttest_done\n+fi\n+\n+ROOT_PATH=\"$PWD\"\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5550'}\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'setup remote repository' '\n+\tcd \"$ROOT_PATH\" &&\n+\tmkdir test_repo &&\n+\tcd test_repo &&\n+\tgit init &&\n+\t: >path1 &&\n+\tgit add path1 &&\n+\ttest_tick &&\n+\tgit commit -m initial &&\n+\tcd - &&\n+\tgit clone --bare test_repo test_repo.git &&\n+\tcd test_repo.git &&\n+\tgit config http.receivepack true &&\n+\tORIG_HEAD=$(git rev-parse --verify HEAD) &&\n+\tcd - &&\n+\tmv test_repo.git \"$HTTPD_DOCUMENT_ROOT_PATH\"\n+'\n+\n+test_expect_success 'clone remote repository' '\n+\tcd \"$ROOT_PATH\" &&\n+\tgit clone $HTTPD_URL/git/test_repo.git test_repo_clone\n+'\n+\n+test_expect_success 'push to remote repository with packed refs' '\n+\tcd \"$ROOT_PATH\"/test_repo_clone &&\n+\t: >path2 &&\n+\tgit add path2 &&\n+\ttest_tick &&\n+\tgit commit -m path2 &&\n+\tHEAD=$(git rev-parse --verify HEAD) &&\n+\tgit push &&\n+\t(cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n+\t test $HEAD = $(git rev-parse --verify HEAD))\n+'\n+\n+test_expect_success 'push already up-to-date' '\n+\tgit push\n+'\n+\n+test_expect_success 'push to remote repository with unpacked refs' '\n+\t(cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n+\t rm packed-refs &&\n+\t git update-ref refs/heads/master $ORIG_HEAD) &&\n+\tgit push &&\n+\t(cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n+\t test $HEAD = $(git rev-parse --verify HEAD))\n+'\n+\n+test_expect_success 'create and delete remote branch' '\n+\tcd \"$ROOT_PATH\"/test_repo_clone &&\n+\tgit checkout -b dev &&\n+\t: >path3 &&\n+\tgit add path3 &&\n+\ttest_tick &&\n+\tgit commit -m dev &&\n+\tgit push origin dev &&\n+\tgit push origin :dev &&\n+\ttest_must_fail git show-ref --verify refs/remotes/origin/dev\n+'\n+\n+cat >exp <act <\"$HTTPD_ROOT_PATH\"/access.log &&\n+\ttest_cmp exp act\n+'\n+\n+stop_httpd\n+test_done\ndiff --git a/t/t5550-http-fetch.sh b/t/t5550-http-fetch.sh\nindex 0e69324..8d6443f 100755\n--- a/t/t5550-http-fetch.sh\n+++ b/t/t5550-http-fetch.sh\n@@ -1,6 +1,6 @@\n #!/bin/sh\n \n-test_description='test fetching over http'\n+test_description='test dumb fetching over http via static file'\n . ./test-lib.sh\n \n if test -n \"$NO_CURL\"; then\n@@ -61,5 +61,11 @@ test_expect_success 'fetch packed objects' '\n \tgit clone $HTTPD_URL/repo_pack.git\n '\n \n+test_expect_success 'did not use upload-pack service' '\n+\tgrep '/git-upload-pack' <\"$HTTPD_ROOT_PATH\"/access.log >act\n+\t: >exp\n+\ttest_cmp exp act\n+'\n+\n stop_httpd\n test_done\ndiff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh\nnew file mode 100755\nindex 0000000..afddc2c\n--- /dev/null\n+++ b/t/t5551-http-fetch.sh\n@@ -0,0 +1,87 @@\n+#!/bin/sh\n+\n+test_description='test smart fetching over http via http-backend'\n+. ./test-lib.sh\n+\n+if test -n \"$NO_CURL\"; then\n+\tsay 'skipping test, git built without http support'\n+\ttest_done\n+fi\n+\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5550'}\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'setup repository' '\n+\techo content >file &&\n+\tgit add file &&\n+\tgit commit -m one\n+'\n+\n+test_expect_success 'create http-accessible bare repository' '\n+\tmkdir \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\t(cd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\t git --bare init\n+\t) &&\n+\tgit remote add public \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit push public master:master\n+'\n+\n+cat >exp < GET /git/repo.git/info/refs?service=git-upload-pack HTTP/1.1\n+Accept: */*\n+Pragma: no-cache\n+< HTTP/1.1 200 OK\n+< Pragma: no-cache\n+< Cache-Control: no-cache, max-age=0, must-revalidate\n+< Content-Type: application/x-git-upload-pack-advertisement\n+< \n+> POST /git/repo.git/git-upload-pack HTTP/1.1\n+Accept-Encoding: deflate, gzip\n+Content-Type: application/x-git-upload-pack-request\n+Accept: application/x-git-upload-pack-response\n+Content-Length: xxxx\n+< HTTP/1.1 200 OK\n+< Pragma: no-cache\n+< Cache-Control: no-cache, max-age=0, must-revalidate\n+< Content-Type: application/x-git-upload-pack-result\n+< \n+EOF\n+test_expect_success 'clone http repository' '\n+\tGIT_CURL_VERBOSE=1 git clone $HTTPD_URL/git/repo.git clone 2>err &&\n+\ttest_cmp file clone/file &&\n+\tegrep \"^([<>]|Pragma|Accept|Content-|Transfer-)\" err |\n+\tegrep -v \"^< (Server|Expires|Date|Content-Length:|Transfer-Encoding: chunked)\" |\n+\tsed -e \"\n+\t\ts/\n//\n+\t\ts/^Content-Length: .*$/Content-Length: xxxx/\n+\t\" >act &&\n+\ttest_cmp exp act\n+'\n+\n+test_expect_success 'fetch changes via http' '\n+\techo content >>file &&\n+\tgit commit -a -m two &&\n+\tgit push public\n+\t(cd clone && git pull) &&\n+\ttest_cmp file clone/file\n+'\n+\n+cat >exp <act <\"$HTTPD_ROOT_PATH\"/access.log &&\n+\ttest_cmp exp act\n+'\n+\n+stop_httpd\n+test_done\n-- \n1.6.5.2.181.gd6f41\n"},{"id":"126225","messageId":"20091029003117.GA18299@localhost","threadId":"21411","inReplyTo":"1256774448-7625-27-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2009-10-29T00:31:17Z","receivedAt":"2009-10-29T00:31:17Z","isPatch":true,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Wed, Oct 28, 2009 at 05:00:48PM -0700, Shawn O. Pearce wrote:\n\n> --- /dev/null\n> +++ b/t/t5541-http-push.sh\n[...]\n> +LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5550'}\n\nThis should be 5541. We need different ports to be able to run the tests\nsimultenously.\n\n> --- a/t/t5550-http-fetch.sh\n> +++ b/t/t5550-http-fetch.sh\n\nThere is also a http port related bug in t5550. I'm attaching the patch\nbelow. Maybe you want to squash this in here.\n\n> --- /dev/null\n> +++ b/t/t5551-http-fetch.sh\n[...]\n> +LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5550'}\n\nDitto, 5551.\n\nClemens\n\n-->o--\nSubject: [PATCH] set httpd port before sourcing lib-httpd\n\nIf LIB_HTTPD_PORT is not set already, lib-httpd will set it to the\ndefault 8111.\n\nSigned-off-by: Clemens Buchacher \n---\n t/t5540-http-push.sh | 7 +++----\n 1 files changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t5540-http-push.sh b/t/t5540-http-push.sh\nindex 5c0f4d7..ea46d1e 100755\n--- a/t/t5540-http-push.sh\n+++ b/t/t5540-http-push.sh\n@@ -9,17 +9,16 @@ This test runs various sanity checks on http-push.'\n \n . ./test-lib.sh\n \n-ROOT_PATH=\"$PWD\"\n-LIB_HTTPD_DAV=t\n-LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5540'}\n-\n if git http-push > /dev/null 2>&1 || [ $? -eq 128 ]\n then\n \tsay \"skipping test, USE_CURL_MULTI is not defined\"\n \ttest_done\n fi\n \n+LIB_HTTPD_DAV=t\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5540'}\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n+ROOT_PATH=\"$PWD\"\n start_httpd\n \n test_expect_success 'setup remote repository' '\n-- \n1.6.5.1.208.gd7b37\n"},{"id":"126229","messageId":"7vpr86dgyj.fsf@alter.siamese.dyndns.org","threadId":"21411","inReplyTo":"1256774448-7625-27-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-29T03:20:04Z","receivedAt":"2009-10-29T03:20:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> +test_expect_success 'clone http repository' '\n> +\tGIT_CURL_VERBOSE=1 git clone $HTTPD_URL/git/repo.git clone 2>err &&\n> +\ttest_cmp file clone/file &&\n> +\tegrep \"^([<>]|Pragma|Accept|Content-|Transfer-)\" err |\n> +\tegrep -v \"^< (Server|Expires|Date|Content-Length:|Transfer-Encoding: chunked)\" |\n> +\tsed -e \"\n> +\t\ts/\n> //\n> +\t\ts/^Content-Length: .*$/Content-Length: xxxx/\n> +\t\" >act &&\n\nThis chomped line is so unlike you---what happened?\n\nAlso, when the last downstream is sed, why would you even need two egrep\nprocess?\n"},{"id":"126230","messageId":"7vljiudgrr.fsf@alter.siamese.dyndns.org","threadId":"21411","inReplyTo":"1256774448-7625-6-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 05/26] Move \"get_ack()\" back to fetch-pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-29T03:24:08Z","receivedAt":"2009-10-29T03:24:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> In 41cb7488 Linus moved this function to connect.c for reuse inside\n> of the git-clone-pack command. That was 2005, but in 2006 Junio\n> retired git-clone-pack in commit efc7fa53. Since then the only\n> caller has been fetch-pack. Since this ACK/NAK exchange is only\n> used by the fetch-pack/upload-pack protocol we should keep move\n> it back to a private detail of fetch-pack.\n\nShould we keep it there or should we move it? which? ;-)\n"},{"id":"126231","messageId":"7vhbtidgmp.fsf@alter.siamese.dyndns.org","threadId":"21411","inReplyTo":"1256774448-7625-4-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-29T03:27:10Z","receivedAt":"2009-10-29T03:27:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> diff --git a/pkt-line.c b/pkt-line.c\n> index bd603f8..893dd3c 100644\n> --- a/pkt-line.c\n> +++ b/pkt-line.c\n> @@ -124,12 +124,14 @@ static int packet_length(const char *linelen)\n> int packet_read_line(int fd, char *buffer, unsigned size)\n> {\n> \tint len;\n> -\tchar linelen[4];\n> +\tchar linelen[5];\n> \n> \tsafe_read(fd, linelen, 4);\n> \tlen = packet_length(linelen);\n> -\tif (len < 0)\n> -\t\tdie(\"protocol error: bad line length character\");\n> +\tif (len < 0) {\n> +\t\tlinelen[4] = '\\0';\n> +\t\tdie(\"protocol error: bad line length character: %s\", linelen);\n> +\t}\n\nSince this is not called recursively, you can make linelen[] static and do\nwithout the NUL assignment; safe_read() won't read beyond 4 bytes anyway.\n\n> \tif (!len)\n> \t\treturn 0;\n> \tlen -= 4;\n> -- \n> 1.6.5.2.181.gd6f41\n"},{"id":"126232","messageId":"7vd446dfx4.fsf@alter.siamese.dyndns.org","threadId":"21411","inReplyTo":"1256774448-7625-15-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 14/26] Add stateless RPC options to upload-pack, receive-pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-29T03:42:31Z","receivedAt":"2009-10-29T03:42:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> When --stateless-rpc is passed as a command line parameter to\n> upload-pack or receive-pack the programs now assume they may\n> perform only a single read-write cycle with stdin and stdout.\n> This fits with the HTTP POST request processing model where a\n> program may read the request, write a response, and must exit.\n>\n> When --advertise-refs is passed as a command line parameter only\n> the initial ref advertisement is output, and the program exits\n> immediately. This fits with the HTTP GET request model, where\n> no request content is received but a response must be produced.\n\nIs the idea to first run with --advertise-refs to get the set of refs, and\nthen doing another, separate run with --stateless-rpc, assuming that the\nrefs the other end advertised does not change in the meantime, to conclude\nthe business?\n\nI suspect that two separate invocations on a (supposedly) single\nrepository made back-to-back can produce an inconsistent response in\nverious situations (e.g. somebody pushing in the middle, or the same\nhostname served by more than one mirrors) and the other end can get\nconfused.\n\nI do not think there is any way to avoid it, short of adding to the second\nrequest some \"cookie\" that allows the second requestee to verify that the\nrequest is based on what he would give to the requester if this request\nwere the first step of the request made to him, iow, his state did not\nchange in the middle since the first request was made (either to him or to\nthe equivalent mirror sitting next to himm).\n\nI wouldn't worry too much about this if the only negative effect could be\nthat the requestor's second request may result in an error, but I am\nwondering if this can be used to attack the requestee.\n"},{"id":"126233","messageId":"7v4opibv4i.fsf@alter.siamese.dyndns.org","threadId":"21411","inReplyTo":"1256774448-7625-7-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 06/26] Add multi_ack_detailed capability to fetch-pack/upload-pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-29T05:57:01Z","receivedAt":"2009-10-29T05:57:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> ACK %s\n> -----------------------------------\n> * no multi_ack or multi_ack_detailed:\n>\n> Sent in response to \"have\" when the object exists on the remote\n> side and is therefore an object in common between the peers.\n> The argument is the SHA-1 of the common object.\n\nDo you mean by \"exists\" something a bit stronger than that, namely, it\nexists and everything reachable from it also exists, right?\n\n> ACK %s common\n> -----------------------------------\n> * multi_ack_detailed only:\n>\n> Sent in response to \"have\". Both sides have this object.\n> Like with \"ACK %s continue\" above the client should stop\n> sending have lines reachable for objects from the argument.\n>\n> ACK %s ready\n> -----------------------------------\n> * multi_ack_detailed only:\n>\n> Sent in response to \"have\".\n>\n> The client should stop transmitting objects which are reachable\n> from the argument, and send \"done\" soon to get the objects.\n>\n> If the remote side has the specified object, it should\n> first send an \"ACK %s common\" message prior to sending\n> \"ACK %s ready\".\n>\n> Clients may still submit additional \"have\" lines if there are\n> more side branches for the client to explore that might be added\n> to the common set and reduce the number of objects to transfer.\n\nI do not understand this after reading it three times. The remote side\nsays \"ACK $it common\", allow the requestor to feed more \"have\", then\neventually send an \"ACK $it ready\" for the same object it earlier sent\n\"common\" for? The first one tells the requestor not to go down the path\nfrom $it further, so presumably all the \"have\"s that come after it will be\nabout different ancestry paths.\n\nWhat is the advantage of using this? In other words, how, in what\nsituation and why would the remote side choose to use \"ready\" --- it looks\nto me that it could always say \"common\" whenever it hits a \"have\" that it\ncan determine to be common.\n"},{"id":"126253","messageId":"m3my3ad3gp.fsf@localhost.localdomain","threadId":"21411","inReplyTo":"1256774448-7625-1-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 00/26] Return of smart HTTP","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-10-29T08:12:17Z","receivedAt":"2009-10-29T08:12:17Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> I think this is the final spin of the smart HTTP series. I've\n> collected patches from a few others (thanks folks!) and added\n> tests specific to the smart variant of the HTTP transport.\n> \n> At this point, I think it is \"next ready\"... but would appreciate\n> any additional feedback if folks identify something we should\n> address before hitting next.\n[...]\n\n> .gitignore | 1 +\n> Documentation/config.txt | 8 +\n> Documentation/git-http-backend.txt | 170 ++++++++\n> Documentation/git-remote-helpers.txt | 85 ++++-\n> Makefile | 1 +\n[...]\n\n> 32 files changed, 2574 insertions(+), 223 deletions(-)\n> create mode 100644 Documentation/git-http-backend.txt\n> create mode 100644 http-backend.c\n> create mode 100755 t/t5541-http-push.sh\n> create mode 100755 t/t5551-http-fetch.sh\n \nIf it is a final spin, then what of missing RFC-like documentation of\nGit HTTP protocol in Documentation/technical/http-protocol.txt\n(it was present only in first version of new series)?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"126272","messageId":"20091029143245.GT10505@spearce.org","threadId":"21411","inReplyTo":"20091029003117.GA18299@localhost","subject":"Re: [RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T14:32:45Z","receivedAt":"2009-10-29T14:32:45Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Clemens Buchacher wrote:\n> On Wed, Oct 28, 2009 at 05:00:48PM -0700, Shawn O. Pearce wrote:\n> \n> > --- /dev/null\n> > +++ b/t/t5541-http-push.sh\n> [...]\n> > +LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5550'}\n> \n> This should be 5541. We need different ports to be able to run the tests\n> simultenously.\n\nThanks, I also realized this myself this morning on my way into work.\nI've adjusted my two new test scripts, and inserted your patch to\nfix the older test script into the series.\n \n-- \nShawn.\n"},{"id":"126273","messageId":"20091029143702.GU10505@spearce.org","threadId":"21411","inReplyTo":"7vpr86dgyj.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T14:37:02Z","receivedAt":"2009-10-29T14:37:02Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > +test_expect_success 'clone http repository' '\n> > +\tGIT_CURL_VERBOSE=1 git clone $HTTPD_URL/git/repo.git clone 2>err &&\n> > +\ttest_cmp file clone/file &&\n> > +\tegrep \"^([<>]|Pragma|Accept|Content-|Transfer-)\" err |\n> > +\tegrep -v \"^< (Server|Expires|Date|Content-Length:|Transfer-Encoding: chunked)\" |\n> > +\tsed -e \"\n> > +\t\ts/\n> > //\n> > +\t\ts/^Content-Length: .*$/Content-Length: xxxx/\n> > +\t\" >act &&\n> \n> This chomped line is so unlike you---what happened?\n\nI was getting different Content-Lengths on different runs of the\ntest. I don't know why. Here the Content-Length is of the gzip'd\nrequest, it shouldn't be varying with each run, but it seemed to be.\n \n> Also, when the last downstream is sed, why would you even need two egrep\n> process?\n\nThat's a really good point. This is just stupid, I started with\nthe two egreps to filter the lines, then found I needed to strip\nCRs, and then had to munge the Content-Length, and I just forgot\nto merge the egrep cases into the sed script.\n\nI'll fix this. Thanks.\n\n-- \nShawn.\n"},{"id":"126275","messageId":"20091029145604.GV10505@spearce.org","threadId":"21411","inReplyTo":"20091029143702.GU10505@spearce.org","subject":"Re: [RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T14:56:04Z","receivedAt":"2009-10-29T14:56:04Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Shawn O. Pearce\" wrote:\n> Junio C Hamano wrote:\n> > > +\t\ts/^Content-Length: .*$/Content-Length: xxxx/\n> > \n> > This chomped line is so unlike you---what happened?\n> \n> I was getting different Content-Lengths on different runs of the\n> test. I don't know why. Here the Content-Length is of the gzip'd\n> request, it shouldn't be varying with each run, but it seemed to be.\n\nFWIW, I found the variablity here. It was triggered by running the\ntest with -v or without -v, which caused clone to change its first\n\"want\" line to include or exclude progress messages, based on the\ntty-ness of stderr.\n\nAdding a --quiet to the clone command means I can avoid this munging\nand hard code the length in the test vector. But I'm still against\ndoing that here in case the protocol capabilities ever change in\nthe future.\n \n-- \nShawn.\n"},{"id":"126276","messageId":"20091029150835.GW10505@spearce.org","threadId":"21411","inReplyTo":"7vljiudgrr.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH v4 05/26] Move \"get_ack()\" back to fetch-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T15:08:35Z","receivedAt":"2009-10-29T15:08:35Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > In 41cb7488 Linus moved this function to connect.c for reuse inside\n> > of the git-clone-pack command. That was 2005, but in 2006 Junio\n> > retired git-clone-pack in commit efc7fa53. Since then the only\n> > caller has been fetch-pack. Since this ACK/NAK exchange is only\n> > used by the fetch-pack/upload-pack protocol we should keep move\n> > it back to a private detail of fetch-pack.\n> \n> Should we keep it there or should we move it? which? ;-)\n\nRight. Fixed.\n\n-- \nShawn.\n"},{"id":"126277","messageId":"20091029151152.GX10505@spearce.org","threadId":"21411","inReplyTo":"7vhbtidgmp.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T15:11:52Z","receivedAt":"2009-10-29T15:11:52Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> > diff --git a/pkt-line.c b/pkt-line.c\n> > index bd603f8..893dd3c 100644\n> > --- a/pkt-line.c\n> > +++ b/pkt-line.c\n> > @@ -124,12 +124,14 @@ static int packet_length(const char *linelen)\n> > int packet_read_line(int fd, char *buffer, unsigned size)\n> > {\n> > \tint len;\n> > -\tchar linelen[4];\n> > +\tchar linelen[5];\n> > \n> > \tsafe_read(fd, linelen, 4);\n> > \tlen = packet_length(linelen);\n> > -\tif (len < 0)\n> > -\t\tdie(\"protocol error: bad line length character\");\n> > +\tif (len < 0) {\n> > +\t\tlinelen[4] = '\\0';\n> > +\t\tdie(\"protocol error: bad line length character: %s\", linelen);\n> > +\t}\n> \n> Since this is not called recursively, you can make linelen[] static\n\nSure. But it wasn't static before. It was stack allocated. Its a\n5 byte stack allocation. Its not a lot of data to push onto the\nstack, why does it suddenly matter that we make it static and charge\neveryone for its memory instead of just those who really need it?\n\n> and do\n> without the NUL assignment; safe_read() won't read beyond 4 bytes anyway.\n\nThe NUL assignment isn't about safe_read(), its about making the\nblock of 4 bytes read a proper C-style string so we can safely pass\nit down into the snprintf that is within die().\n\n-- \nShawn.\n"},{"id":"126278","messageId":"20091029152629.GY10505@spearce.org","threadId":"21411","inReplyTo":"7vd446dfx4.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH v4 14/26] Add stateless RPC options to upload-pack, receive-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T15:26:29Z","receivedAt":"2009-10-29T15:26:29Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > When --stateless-rpc is passed [...]\n> > When --advertise-refs is passed [...]\n> \n> Is the idea to first run with --advertise-refs to get the set of refs, and\n> then doing another, separate run with --stateless-rpc, assuming that the\n> refs the other end advertised does not change in the meantime, to conclude\n> the business?\n\nYes.\n \n> I suspect that two separate invocations on a (supposedly) single\n> repository made back-to-back can produce an inconsistent response in\n> verious situations (e.g. somebody pushing in the middle, or the same\n> hostname served by more than one mirrors) and the other end can get\n> confused.\n\nYes, that can happen.\n\n> I do not think there is any way to avoid it, short of adding to the second\n> request some \"cookie\" that allows the second requestee to verify that the\n> request is based on what he would give to the requester if this request\n> were the first step of the request made to him, iow, his state did not\n> change in the middle since the first request was made (either to him or to\n> the equivalent mirror sitting next to himm).\n\nThere isn't a lot we can do, you are right.\n\nOne approach is to use an HMAC and sign each advertised SHA-1\nduring the initial --advertise-refs phase. Requesters then present\nthe SHA-1 and the HAMC signature in each \"want\" line, and the\n--stateless-rpc phase validates the signatures to ensure they came\nfrom a trusted party.\n\nThe major problem with this approach is the private key management.\nAll mirrors of that repository need to have a common private key\nso they can generate and later verify that HMAC signature. This is\nadditional complexity, for perhaps not much gain.\n\nA different approach is to have the --stateless-rpc phase validate\nthe want lines against its refs (like we do now), but if the validate\nfails (no ref exactly matches) support walking backwards through the\nlast few records in the reflog, or through that ref's own history\nfor a few hundred commits, to see if the want SHA-1 appears in the\nrecent prior past.\n\nObviously when walking the reflog we would need to use a time bound,\ne.g. \"only examine last record if more recent than 5 minutes ago\".\nThis way a force push to erase something will still erase it, but\na client who had just recently observed the prior SHA-1 can still\ncomplete their fetch request, just as with native git:// they could\ndo that due to the prior SHA-1 being held in the git-upload-pack's\nprivate memory.\n\nAlso, obviously when walking the commit history of a ref (to see\nif the want'd SHA-1 is merged into one or more reachable refs)\nwe don't want to walk backwards too far, as it costs CPU time on\nthe server side, but we also don't want to walk too few commits.\nE.g. pushes for me tend to be in the 20 commit range, while Linus\nprobably pushes a thousand commits or more in a single merge.\nFinding the right balance may be tricky.\n\n> I wouldn't worry too much about this if the only negative effect could be\n> that the requestor's second request may result in an error, but I am\n> wondering if this can be used to attack the requestee.\n\nI don't think it can be used to attack the server. The only impact\nI can see is the client gets confused and gets an error response\nfrom the server when the server aborts due to the invalid \"want\"\nline sent during that 2nd (or any subsequent) request.\n\n-- \nShawn.\n"},{"id":"126280","messageId":"20091029152839.GZ10505@spearce.org","threadId":"21411","inReplyTo":"m3my3ad3gp.fsf@localhost.localdomain","subject":"Re: [RFC PATCH v4 00/26] Return of smart HTTP","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T15:28:39Z","receivedAt":"2009-10-29T15:28:39Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > I think this is the final spin of the smart HTTP series.\n> \n> If it is a final spin, then what of missing RFC-like documentation of\n> Git HTTP protocol in Documentation/technical/http-protocol.txt\n> (it was present only in first version of new series)?\n\nIts still incomplete. I want to get the code cooking while I work\non the documentation. Its a separate patch anyway, whether or not\nits in the middle of the code series or after it is doesn't matter.\n\n-- \nShawn.\n"},{"id":"126287","messageId":"20091029161748.GB10505@spearce.org","threadId":"21411","inReplyTo":"7v4opibv4i.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH v4 06/26] Add multi_ack_detailed capability to fetch-pack/upload-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T16:17:48Z","receivedAt":"2009-10-29T16:17:48Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > ACK %s\n> > -----------------------------------\n> > * no multi_ack or multi_ack_detailed:\n> >\n> > Sent in response to \"have\" when the object exists on the remote\n> > side and is therefore an object in common between the peers.\n> > The argument is the SHA-1 of the common object.\n> \n> Do you mean by \"exists\" something a bit stronger than that, namely, it\n> exists and everything reachable from it also exists, right?\n\nNo, I mean \"object exists\". The upload-pack code makes no check that\nthe object is in fact complete, just that it has the object in the\nobject database. The object could be a dangling commit whose parents\naren't present, and we'd still (currently) return an \"ACK %s\" for it.\n\nI specifically didn't get into completeness here because we don't\nactually check for it.\n \n> > ACK %s common\n> > -----------------------------------\n> > * multi_ack_detailed only:\n> >\n> > Sent in response to \"have\". Both sides have this object.\n> > Like with \"ACK %s continue\" above the client should stop\n> > sending have lines reachable for objects from the argument.\n> >\n> > ACK %s ready\n> > -----------------------------------\n> > * multi_ack_detailed only:\n> >\n> > Sent in response to \"have\".\n> >\n> > The client should stop transmitting objects which are reachable\n> > from the argument, and send \"done\" soon to get the objects.\n> >\n> > If the remote side has the specified object, it should\n> > first send an \"ACK %s common\" message prior to sending\n> > \"ACK %s ready\".\n> >\n> > Clients may still submit additional \"have\" lines if there are\n> > more side branches for the client to explore that might be added\n> > to the common set and reduce the number of objects to transfer.\n> \n> I do not understand this after reading it three times. The remote side\n> says \"ACK $it common\", allow the requestor to feed more \"have\", then\n> eventually send an \"ACK $it ready\" for the same object it earlier sent\n> \"common\" for? The first one tells the requestor not to go down the path\n> from $it further, so presumably all the \"have\"s that come after it will be\n> about different ancestry paths.\n> \n> What is the advantage of using this? In other words, how, in what\n> situation and why would the remote side choose to use \"ready\" --- it looks\n> to me that it could always say \"common\" whenever it hits a \"have\" that it\n> can determine to be common.\n\n\"ACK $it ready\" should be used when ok_to_give_up() returns true.\n\nThe \"ACK $it ready\" message is trying to say \"don't talk any more\nabout things reachable from $it, and by the way, if you say 'done'\nnow I will give you a pack\".\n\nThe \"ACK $it common\" message is trying to say \"$it is a common base,\ndon't talk any more about things reachable from $it, but there\nmay be other branches you should explore before I send you a pack,\nso talk about those if you can\".\n\n\nA client only wants to store $it into its request state vector for\nreplay on the next RPC if $it is truely common.\n\nThe \"ACK $it common\" before \"ACK $it ready\" is because clients can't\nassume that \"ACK $it ready\" means $it is really common. Servers send\n\"ACK $it ready\" if they don't have $it but are ok_to_give_up().\nBut if it is common, *and* ok_to_give_up() is true, a server can\nsend both messages, but it must send \"ACK $it common\" first.\n\n-- \nShawn.\n"},{"id":"126330","messageId":"7v1vkm6id9.fsf@alter.siamese.dyndns.org","threadId":"21411","inReplyTo":"20091029151152.GX10505@spearce.org","subject":"Re: [RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-29T20:43:46Z","receivedAt":"2009-10-29T20:43:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Junio C Hamano wrote:\n>> \"Shawn O. Pearce\" writes:\n>> > diff --git a/pkt-line.c b/pkt-line.c\n>> > index bd603f8..893dd3c 100644\n>> > --- a/pkt-line.c\n>> > +++ b/pkt-line.c\n>> > @@ -124,12 +124,14 @@ static int packet_length(const char *linelen)\n>> > int packet_read_line(int fd, char *buffer, unsigned size)\n>> > {\n>> > \tint len;\n>> > -\tchar linelen[4];\n>> > +\tchar linelen[5];\n>> > \n>> > \tsafe_read(fd, linelen, 4);\n>> > \tlen = packet_length(linelen);\n>> > -\tif (len < 0)\n>> > -\t\tdie(\"protocol error: bad line length character\");\n>> > +\tif (len < 0) {\n>> > +\t\tlinelen[4] = '\\0';\n>> > +\t\tdie(\"protocol error: bad line length character: %s\", linelen);\n>> > +\t}\n>> \n>> Since this is not called recursively, you can make linelen[] static\n>\n> Sure. But it wasn't static before. It was stack allocated. Its a\n> 5 byte stack allocation. Its not a lot of data to push onto the\n> stack, why does it suddenly matter that we make it static and charge\n> everyone for its memory instead of just those who really need it?\n>\n>> and do\n>> without the NUL assignment; safe_read() won't read beyond 4 bytes anyway.\n>\n> The NUL assignment isn't about safe_read(), its about making the\n> block of 4 bytes read a proper C-style string so we can safely pass\n> it down into the snprintf that is within die().\n\nI knew and understood all of what you just said. static linelen[] is not\nabout stack allocation. It's about letting the compiler initialize it to\nfive NULs so that you do not have to assign NUL to its fifth place before\nyou die. This removes one added line from your patch.\n"},{"id":"126331","messageId":"7vtyxi53sh.fsf@alter.siamese.dyndns.org","threadId":"21411","inReplyTo":"20091029152629.GY10505@spearce.org","subject":"Re: [RFC PATCH v4 14/26] Add stateless RPC options to upload-pack, receive-pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-29T20:43:58Z","receivedAt":"2009-10-29T20:43:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> One approach is to use an HMAC and sign each advertised SHA-1\n> during the initial --advertise-refs phase. Requesters then present\n> the SHA-1 and the HAMC signature in each \"want\" line, and the\n> --stateless-rpc phase validates the signatures to ensure they came\n> from a trusted party.\n>\n> The major problem with this approach is the private key management.\n> All mirrors of that repository need to have a common private key\n> so they can generate and later verify that HMAC signature. This is\n> additional complexity, for perhaps not much gain.\n\nI am not worried so much about malicious clients getting themselves\nconfused. One simple alternative would be to internally recreate the\nresponse the requestee would send if it were the first phase request upon\nreceiving the request for the second phase---then you take an SHA-1 hash\nof it. In the second phase request have the requestor include the SHA-1\nhash of what it received in the response to its first phase request, and\nthe requestee can make sure they match. No per-ref signing nor secret key\nmanagement is necessary, and it would let the requestor retry if you allow\nthe response to the second phase request to be \"your request is stale, try\nagain\".\n\n> A different approach is to have the --stateless-rpc phase validate\n> the want lines against its refs (like we do now), but if the validate\n> fails (no ref exactly matches) support walking backwards through the\n> last few records in the reflog, or through that ref's own history\n> for a few hundred commits, to see if the want SHA-1 appears in the\n> recent prior past.\n>\n> Obviously when walking the reflog we would need to use a time bound,\n> e.g. \"only examine last record if more recent than 5 minutes ago\".\n\nI think that is probably too much complexity for too little gain. I think\ndetecting stale request and having requestor retry would be sufficient,\nand validating the want lines as we already do would give the same level\nof assurance as \"check against the hash of first phase response\" I\noutlined above, and would be much simpler thus more robust.\n"},{"id":"126336","messageId":"20091029215829.GD10505@spearce.org","threadId":"21411","inReplyTo":"7v1vkm6id9.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-29T21:58:29Z","receivedAt":"2009-10-29T21:58:29Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> > The NUL assignment isn't about safe_read(), its about making the\n> > block of 4 bytes read a proper C-style string so we can safely pass\n> > it down into the snprintf that is within die().\n> \n> I knew and understood all of what you just said. static linelen[] is not\n> about stack allocation. It's about letting the compiler initialize it to\n> five NULs so that you do not have to assign NUL to its fifth place before\n> you die. This removes one added line from your patch.\n\nThanks, I get it now. Patch amended.\n\n-- \nShawn.\n"},{"id":"126386","messageId":"be6fef0d0910300802t44801cb9n7a129905725f5c56@mail.gmail.com","threadId":"21411","inReplyTo":"1256774448-7625-12-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 11/26] Move WebDAV HTTP push under remote-curl","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2009-10-30T15:02:39Z","receivedAt":"2009-10-30T15:02:39Z","isPatch":true,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Thu, Oct 29, 2009 at 8:00 AM, Shawn O. Pearce wrote:\n>  update http tests according to remote-curl capabilities\n\nit would be great if you could mention the $ORIG_HEAD bit:\n\n o Use a variable ($ORIG_HEAD) instead of full SHA-1 name.\n\n-- \nCheers,\nRay Chuan\n"},{"id":"126392","messageId":"be6fef0d0910300910me43c77fue6dcb6034dd0ea5b@mail.gmail.com","threadId":"21411","inReplyTo":"1256774448-7625-12-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 11/26] Move WebDAV HTTP push under remote-curl","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2009-10-30T16:10:29Z","receivedAt":"2009-10-30T16:10:29Z","isPatch":true,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Thu, Oct 29, 2009 at 8:00 AM, Shawn O. Pearce wrote:\n> diff --git a/t/t5540-http-push.sh b/t/t5540-http-push.sh\n>[snip]\n> @@ -57,11 +58,15 @@ test_expect_failure 'push to remote repository with packed refs' '\n>         test $HEAD = $(git rev-parse --verify HEAD))\n>  '\n>\n> -test_expect_success ' push to remote repository with unpacked refs' '\n> +test_expect_failure 'push already up-to-date' '\n> +       git push\n> +'\n> +\n> +test_expect_success 'push to remote repository with unpacked refs' '\n>        (cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n>         rm packed-refs &&\n> -        git update-ref refs/heads/master \\\n> -               0c973ae9bd51902a28466f3850b543fa66a6aaf4) &&\n> +        git update-ref refs/heads/master $ORIG_HEAD &&\n> +        git --bare update-server-info) &&\n>        git push &&\n>        (cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n>         test $HEAD = $(git rev-parse --verify HEAD))\n\nClemens, the following addresses your non-desire to remove the\n\"unpacked refs\" test in your earlier email\n<20091025161630.GB8532@localhost>.\n\nNow that the first push in \"push to remote repository with packed\nrefs\" succeeds, the \"unpacked refs\" test is redundant. Since http-push\nin that first test already updated /refs/heads/master and /info/refs,\n'git update-ref' incorrectly reverts the earlier push, and 'git\nupdate-server-info' is redundant.\n\n-- \nCheers,\nRay Chuan\n"},{"id":"126393","messageId":"be6fef0d0910300910x5e8bc552k4f020ca8bb890352@mail.gmail.com","threadId":"21411","inReplyTo":"1256774448-7625-27-git-send-email-spearce@spearce.org","subject":"Re: [RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2009-10-30T16:10:45Z","receivedAt":"2009-10-30T16:10:45Z","isPatch":true,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Thu, Oct 29, 2009 at 8:00 AM, Shawn O. Pearce wrote:\n> The top level directory \"/git/\" of the test Apache server is mapped\n> through our git-http-backend CGI, but uses the same underlying\n> repository space as the server's document root. This is the most\n> simple installation possible.\n\nHaving \"/git/\" reside as a subdirectory in \"/\" where WebDAV is enabled\nmay be confusing to readers. I think we should use \"/smart/\" for the\nCGI map, and consequently, use \"/dumb/\" for WebDAV repositories,\nrather than the root \"/\" that it is occupying.\n\n> diff --git a/t/t5541-http-push.sh b/t/t5541-http-push.sh\n>[snip]\n> +test_expect_success 'push to remote repository with packed refs' '\n> + cd \"$ROOT_PATH\"/test_repo_clone &&\n> + : >path2 &&\n> + git add path2 &&\n> + test_tick &&\n> + git commit -m path2 &&\n> + HEAD=$(git rev-parse --verify HEAD) &&\n> + git push &&\n> + (cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n> + test $HEAD = $(git rev-parse --verify HEAD))\n> +'\n> +\n> +test_expect_success 'push already up-to-date' '\n> + git push\n> +'\n> +\n> +test_expect_success 'push to remote repository with unpacked refs' '\n> + (cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n> + rm packed-refs &&\n> + git update-ref refs/heads/master $ORIG_HEAD) &&\n> + git push &&\n> + (cd \"$HTTPD_DOCUMENT_ROOT_PATH\"/test_repo.git &&\n> + test $HEAD = $(git rev-parse --verify HEAD))\n> +'\n\nMention of \"packed refs\" should be removed from the description, and\nthe 'unpacked refs' test, irrelevant in this context, should be\nremoved too. The assumptions these tests are based on is relevant to\nt5540, but not in t5541. My explanation follows.\n\n(Clemens, the following also addresses your non-desire to remove the\n\"unpacked refs\" test in your earlier email\n<20091025161630.GB8532@localhost>.)\n\nIn the \"old\" (v1.6.5) http push mechanism, no refspec is passed to the\nhttp-push helper. This shouldn't be the case, because match_refs is\ndone in transport.c::transport_push, but then transport->push_refs\nisn't defined so this doesn't happen.\n\nhttp-push is then depended on to learn of the remote refs and match\nthem itself, but it does badly at this, since it only recurses through\n/refs/heads in the remote repository. None could be found, since\nthey've all been packed. Thus the push in the first test failed.\nNothing has been pushed to the remote repository. The following push\nin the \"unpacked refs\" corrects this after \"unpacking\" the refs.\n\nClearly, these test are about the ability of http push mechanism to\nlearn of refs in /refs/heads and (lack thereof) from /packed-refs.\n\nBut in this patch series, this is no longer the case.\ntransport->push_refs is now defined, so what happens is that the\nhttp-push helper is passed a refspec, unlike in the \"old\" mechanism.\nhttp-push, using this refspec, now matches refs properly (though it\nstill does its recursion thing). Thus the push in the first test\n(should) succeed. The push in the \"unpacked refs\" test is now\nirrelevant, since the first push has already successfully pushed\nchanges to the remote repository.\n\nSo, what we should have is just one no-frills push test, keeping as\nwell the \"push already up-to-date\" test.\n\n-- \nCheers,\nRay Chuan\n"},{"id":"126400","messageId":"20091030175741.GC18583@coredump.intra.peff.net","threadId":"21411","inReplyTo":"20091029215829.GD10505@spearce.org","subject":"Re: [RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-10-30T17:57:41Z","receivedAt":"2009-10-30T17:57:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Oct 29, 2009 at 02:58:29PM -0700, Shawn O. Pearce wrote:\n\n> Junio C Hamano wrote:\n> > > The NUL assignment isn't about safe_read(), its about making the\n> > > block of 4 bytes read a proper C-style string so we can safely pass\n> > > it down into the snprintf that is within die().\n> > \n> > I knew and understood all of what you just said. static linelen[] is not\n> > about stack allocation. It's about letting the compiler initialize it to\n> > five NULs so that you do not have to assign NUL to its fifth place before\n> > you die. This removes one added line from your patch.\n> \n> Thanks, I get it now. Patch amended.\n\nI am just a bystander, so maybe my opinion is not worth anything, but\npersonally I think you are obfuscating the code to save a single line.\nWhen I see a static variable, I assume it is because the value should be\nsaved from invocation to invocation, and now I will spend time wondering\nwhy that would be the case here.\n\nIf you really just want to initialize to zero, using\n\n char linelen[5] = { 0 };\n\nshould be sufficient (I think all of the compilers we care about support\nsuch incomplete array assignments, right?).\n\nI think it would be even more clear to leave it as\n\n char linelen[4];\n\nwhich declares how the data is _actually_ expected to be used, and then\ndo:\n\n die(\"protocol error: bad line length character: %.4s\", linelen);\n\n-Peff\n\nPS All of the proposals also suffer from the fact that die will truncate\nbroken output at a NUL sent by the remote. Probably OK, since we are\nexpecting ascii hex or some variant, and if we don't correctly print\nwhat the remote said in a totally broken NUL-filled case, it is not that\nbig a deal.\n"},{"id":"126410","messageId":"20091030190655.GA7442@localhost","threadId":"21411","inReplyTo":"be6fef0d0910300910me43c77fue6dcb6034dd0ea5b@mail.gmail.com","subject":"Re: [RFC PATCH v4 11/26] Move WebDAV HTTP push under remote-curl","fromName":"Clemens Buchacher","fromEmail":"drizzd@aon.at","sentAt":"2009-10-30T19:06:55Z","receivedAt":"2009-10-30T19:06:55Z","isPatch":true,"sender":{"key":"drizzd@gmx.net","avatar":"https://avatars.githubusercontent.com/u/59082?v=4"},"body":"On Sat, Oct 31, 2009 at 12:10:29AM +0800, Tay Ray Chuan wrote:\n\n> Clemens, the following addresses your non-desire to remove the\n> \"unpacked refs\" test in your earlier email\n> <20091025161630.GB8532@localhost>.\n\n> Now that the first push in \"push to remote repository with packed\n> refs\" succeeds, the \"unpacked refs\" test is redundant.\n\nHow can the changed result of one test suddenly make another test redundant?\nThe two are testing different things.\n\n> Since http-push\n> in that first test already updated /refs/heads/master and /info/refs,\n> 'git update-ref' incorrectly reverts the earlier push, and 'git\n> update-server-info' is redundant.\n\nNo, 'git update-ref' correctly reverts the earlier push, so we can push again\nand 'git update-server-info' is therefore necessary for the test to work\nindependently of its predecessors result.\n\nClemens\n"},{"id":"126411","messageId":"20091030191239.GF10505@spearce.org","threadId":"21411","inReplyTo":"20091030175741.GC18583@coredump.intra.peff.net","subject":"Re: [RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-30T19:12:40Z","receivedAt":"2009-10-30T19:12:40Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jeff King wrote:\n> On Thu, Oct 29, 2009 at 02:58:29PM -0700, Shawn O. Pearce wrote:\n> > Junio C Hamano wrote:\n> > > I knew and understood all of what you just said. static linelen[] is not\n> > > about stack allocation. It's about letting the compiler initialize it to\n> > > five NULs so that you do not have to assign NUL to its fifth place before\n> > > you die. This removes one added line from your patch.\n> \n> I am just a bystander, so maybe my opinion is not worth anything, but\n> personally I think you are obfuscating the code to save a single line.\n\nYup, me too. But I'm also willing to do what I need to get my\npatches included in git.git. Smart HTTP is something a lot of people\nhave been waiting for. If the maintainer wants the bikeshed to be\na particular shade of red, I'll paint it that way.\n\n> When I see a static variable, I assume it is because the value should be\n> saved from invocation to invocation, and now I will spend time wondering\n> why that would be the case here.\n\nMe too. I also can't grok the code I just wrote, for the same\nreason, it just reads wrong. But who am I to argue with the guy\nwho is most likely going to be the one who needs to deal with it\nin the future?\n \n> If you really just want to initialize to zero, using\n> \n> char linelen[5] = { 0 };\n\nBleh, I find that has hard to grok as what we have now. Perhaps my\nunderstanding of the relevant standards is incomplete, but I'd read\nthat as linelen[0] = 0, but the other 4 positions are undefined\nand may be not be initialized.\n \n> I think it would be even more clear to leave it as\n> \n> die(\"protocol error: bad line length character: %.4s\", linelen);\n\nI actually considered this one, but again I wasn't clear what would\nhappen in the standard C library when we fed a string that wasn't\nactually NUL terminated. Is the library permitted to call strlen()\nbefore formatting? If so strlen() could SIGSEGV if we are unlucky\nand no NUL is present between our string and the end of the assigned\nmemory region.\n\nTo me, my original version was the most clear, to me and anyone\nelse who could ever possibly come by to read it. The \"one extra\nline of code\" is also only in an error condition which never occurs\n(but did once due to a bug in the HTTP code, which is why I added\nthis patch to my series, to help debug it). Its not like this is\na performance sensitive section of git that Linus is going to come\nback and overhaul.\n\n-- \nShawn.\n"},{"id":"126413","messageId":"20091030193030.GA10985@coredump.intra.peff.net","threadId":"21411","inReplyTo":"20091030191239.GF10505@spearce.org","subject":"Re: [RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-10-30T19:30:30Z","receivedAt":"2009-10-30T19:30:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Oct 30, 2009 at 12:12:40PM -0700, Shawn O. Pearce wrote:\n\n> > I am just a bystander, so maybe my opinion is not worth anything, but\n> > personally I think you are obfuscating the code to save a single line.\n> \n> Yup, me too. But I'm also willing to do what I need to get my\n> patches included in git.git. Smart HTTP is something a lot of people\n> have been waiting for. If the maintainer wants the bikeshed to be\n> a particular shade of red, I'll paint it that way.\n\nYes, I too want it merged ASAP, so please Junio if you disagree with my\ncomments, ignore them. But I did want to point out that the proposed\n\"improvement\" was making things worse, IMHO. So I feel like I was at\nleast bikeshedding his bikeshed, instead of starting my own. ;)\n\n> > If you really just want to initialize to zero, using\n> > \n> > char linelen[5] = { 0 };\n> \n> Bleh, I find that has hard to grok as what we have now. Perhaps my\n> understanding of the relevant standards is incomplete, but I'd read\n> that as linelen[0] = 0, but the other 4 positions are undefined\n> and may be not be initialized.\n\nYour understanding is incomplete. :) C99, 6.7.8, paragraph 19:\n\n ... all subobjects that are not initialized explicitly shall be\n initialized implicitly the same as objects that have static storage\n duration.\n\nI don't recall offhand whether that was the case in C89 or not.\n\nBut that being said, it was an attempt to make the meaning clear. If\nit's not to you (who I consider at least reasonably competent ;) ), then\nit failed.\n\n> > die(\"protocol error: bad line length character: %.4s\", linelen);\n> \n> I actually considered this one, but again I wasn't clear what would\n> happen in the standard C library when we fed a string that wasn't\n> actually NUL terminated. Is the library permitted to call strlen()\n> before formatting? If so strlen() could SIGSEGV if we are unlucky\n> and no NUL is present between our string and the end of the assigned\n> memory region.\n\nNo, using a precision specifier for a string with no NUL is explicitly\nallowed by the standard (and we use it elsewhere in git, IIRC).\n\n> To me, my original version was the most clear, to me and anyone\n> else who could ever possibly come by to read it. The \"one extra\n> line of code\" is also only in an error condition which never occurs\n> (but did once due to a bug in the HTTP code, which is why I added\n> this patch to my series, to help debug it). Its not like this is\n> a performance sensitive section of git that Linus is going to come\n> back and overhaul.\n\nHeh. You never know what performance item Linus will complain about. ;)\nBut yes, I am fine with your initial one, too. The drawback to it (and\nmy \"%.4s\") is not that one line of code is so unbearable, but that it is\none extra thing anyone using it as a string must do, and if they forget,\nwe get a horrible segfault.\n\nAnyway, my complaint has been lodged. Junio is aware of the alternatives\nand can pick whichever he wants.\n\n-Peff\n"},{"id":"126414","messageId":"20091030193108.GA30397@huya.quicinc.com","threadId":"21411","inReplyTo":"20091030191239.GF10505@spearce.org","subject":"Re: [RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"David Brown","fromEmail":"davidb@quicinc.com","sentAt":"2009-10-30T19:31:08Z","receivedAt":"2009-10-30T19:31:08Z","isPatch":true,"sender":{"key":"git@davidb.org","avatar":"https://gravatar.com/avatar/94c86a2938470a74c2eac5e2b69afc0871f79a660295c02219597aba8cb101c1?d=mp&s=160"},"body":"On Fri, Oct 30, 2009 at 12:12:40PM -0700, Shawn O. Pearce wrote:\n\n> > If you really just want to initialize to zero, using\n> > \n> > char linelen[5] = { 0 };\n> \n> Bleh, I find that has hard to grok as what we have now. Perhaps my\n> understanding of the relevant standards is incomplete, but I'd read\n> that as linelen[0] = 0, but the other 4 positions are undefined\n> and may be not be initialized.\n\nC isn't that into undefined things. In this particular case, all\nremaining values will be initialized to zero.\n\n> I actually considered this one, but again I wasn't clear what would\n> happen in the standard C library when we fed a string that wasn't\n> actually NUL terminated. Is the library permitted to call strlen()\n> before formatting? If so strlen() could SIGSEGV if we are unlucky\n> and no NUL is present between our string and the end of the assigned\n> memory region.\n\nThe linux manpage says \"If a precision is given, no null byte\nneed be present\". This text is copied verbatim out of the Posix\nspecification, so a C library that failed to handle this would be\nnon-compliant.\n\nI think the %.4s is clearest, and is, in fact, somewhat\nidiomatic.\n\nDavid\n"},{"id":"126415","messageId":"7veioklloq.fsf@alter.siamese.dyndns.org","threadId":"21411","inReplyTo":"20091030175741.GC18583@coredump.intra.peff.net","subject":"Re: [RFC PATCH v4 03/26] pkt-line: Make packet_read_line easier to debug","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-30T19:35:17Z","receivedAt":"2009-10-30T19:35:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Thu, Oct 29, 2009 at 02:58:29PM -0700, Shawn O. Pearce wrote:\n>\n>> Junio C Hamano wrote:\n>> > > The NUL assignment isn't about safe_read(), its about making the\n>> > > block of 4 bytes read a proper C-style string so we can safely pass\n>> > > it down into the snprintf that is within die().\n>> > \n>> > I knew and understood all of what you just said. static linelen[] is not\n>> > about stack allocation. It's about letting the compiler initialize it to\n>> > five NULs so that you do not have to assign NUL to its fifth place before\n>> > you die. This removes one added line from your patch.\n>> \n>> Thanks, I get it now. Patch amended.\n>\n> I am just a bystander, so maybe my opinion is not worth anything, but\n> personally I think you are obfuscating the code to save a single line.\n\nWell, the comment was not a serious \"you should do it this way\" to begin\nwith. An extra assignment in the error path is not a big deal.\n"},{"id":"126446","messageId":"7vljish69m.fsf@alter.siamese.dyndns.org","threadId":"21411","inReplyTo":"20091029143702.GU10505@spearce.org","subject":"Re: [RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-10-30T22:21:57Z","receivedAt":"2009-10-30T22:21:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Junio C Hamano wrote:\n>> \"Shawn O. Pearce\" writes:\n>> \n>> > +test_expect_success 'clone http repository' '\n>> > +\tGIT_CURL_VERBOSE=1 git clone $HTTPD_URL/git/repo.git clone 2>err &&\n>> > +\ttest_cmp file clone/file &&\n>> > +\tegrep \"^([<>]|Pragma|Accept|Content-|Transfer-)\" err |\n>> > +\tegrep -v \"^< (Server|Expires|Date|Content-Length:|Transfer-Encoding: chunked)\" |\n>> > +\tsed -e \"\n>> > +\t\ts/\n>> > //\n>> > +\t\ts/^Content-Length: .*$/Content-Length: xxxx/\n>> > +\t\" >act &&\n>> \n>> This chomped line is so unlike you---what happened?\n>\n> I was getting different Content-Lengths on different runs of the\n> test. I don't know why. Here the Content-Length is of the gzip'd\n> request, it shouldn't be varying with each run, but it seemed to be.\n\nWhat I meant was the patch corruption. I couldn't tell what you were\ntrying to filter with the first substitution you are giving to sed.\n"},{"id":"126452","messageId":"20091030225910.GG10505@spearce.org","threadId":"21411","inReplyTo":"7vljish69m.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-30T22:59:11Z","receivedAt":"2009-10-30T22:59:11Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> > Junio C Hamano wrote:\n> >> \"Shawn O. Pearce\" writes:\n> >> \n> >> > +\tsed -e \"\n> >> > +\t\ts/\n> >> > //\n> >> \n> >> This chomped line is so unlike you---what happened?\n> \n> What I meant was the patch corruption. I couldn't tell what you were\n> trying to filter with the first substitution you are giving to sed.\n\nOh, that. Its a literal CR on the search side, nothing on the\nreplace side.\n\n-- \nShawn.\n"},{"id":"126455","messageId":"be6fef0d0910301620x2e80864ft67b7d067b00005f2@mail.gmail.com","threadId":"21411","inReplyTo":"20091030190655.GA7442@localhost","subject":"Re: [RFC PATCH v4 11/26] Move WebDAV HTTP push under remote-curl","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2009-10-30T23:20:03Z","receivedAt":"2009-10-30T23:20:03Z","isPatch":true,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Sat, Oct 31, 2009 at 3:06 AM, Clemens Buchacher wrote:\n> How can the changed result of one test suddenly make another test redundant?\n> The two are testing different things.\n>[snip]\n> No, 'git update-ref' correctly reverts the earlier push, so we can push again\n> and 'git update-server-info' is therefore necessary for the test to work\n> independently of its predecessors result.\n\nI said \"redundant\" because in the first push, we've already 1) updated\n/refs/heads/master (which is what the 'git update-ref' here does) and\n2) updated /info/refs (which is what the 'git update-server-info' does\nand 3) pushed changes. To me, the \"unpacked refs\" test reads as a\ntrivial test of http-push's ability to update /refs/heads/master and\n/info/refs (something we've already done in the first \"packed refs\"\ntest), rather than its pushing per se. Is it your intention to keep\nthe \"unpacked refs\" test due to the former?\n\n-- \nCheers,\nRay Chuan\n"},{"id":"126458","messageId":"20091030235936.GH10505@spearce.org","threadId":"21411","inReplyTo":"7vtyxi53sh.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC PATCH v4 14/26] Add stateless RPC options to upload-pack, receive-pack","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-30T23:59:36Z","receivedAt":"2009-10-30T23:59:36Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> I think that is probably too much complexity for too little gain. I think\n> detecting stale request and having requestor retry would be sufficient,\n> and validating the want lines as we already do would give the same level\n> of assurance as \"check against the hash of first phase response\" I\n> outlined above, and would be much simpler thus more robust.\n\nAck.\n\nI think what we want here is to just add an \"ERR invalid want\"\nmessage just before disconnecting when the client gives us a SHA-1\nwhich we don't point to directly. Client implementations can choose\nhow to handle this error. They could retry from the beginning,\nor they could abort.\n\n-- \nShawn.\n"},{"id":"126459","messageId":"20091031000933.GJ10505@spearce.org","threadId":"21411","inReplyTo":"be6fef0d0910300802t44801cb9n7a129905725f5c56@mail.gmail.com","subject":"Re: [RFC PATCH v4 11/26] Move WebDAV HTTP push under remote-curl","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-31T00:09:33Z","receivedAt":"2009-10-31T00:09:33Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Tay Ray Chuan wrote:\n> On Thu, Oct 29, 2009 at 8:00 AM, Shawn O. Pearce wrote:\n> > ?update http tests according to remote-curl capabilities\n> \n> it would be great if you could mention the $ORIG_HEAD bit:\n> \n> o Use a variable ($ORIG_HEAD) instead of full SHA-1 name.\n\nThanks, added.\n\n-- \nShawn.\n"},{"id":"126461","messageId":"20091031001314.GK10505@spearce.org","threadId":"21411","inReplyTo":"be6fef0d0910300910x5e8bc552k4f020ca8bb890352@mail.gmail.com","subject":"Re: [RFC PATCH v4 26/26] test smart http fetch and push","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-10-31T00:13:14Z","receivedAt":"2009-10-31T00:13:14Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Tay Ray Chuan wrote:\n> \n> Having \"/git/\" reside as a subdirectory in \"/\" where WebDAV is enabled\n> may be confusing to readers. I think we should use \"/smart/\" for the\n> CGI map, and consequently, use \"/dumb/\" for WebDAV repositories,\n> rather than the root \"/\" that it is occupying.\n\nYup, great idea. Patch added to switch to /dumb/ for WebDAV,\nand changed to use /smart/ for this patch.\n \n> Mention of \"packed refs\" should be removed from the description, and\n> the 'unpacked refs' test, irrelevant in this context, should be\n> removed too. The assumptions these tests are based on is relevant to\n> t5540, but not in t5541. My explanation follows.\n\nYup, you are right, thanks. Fixed.\n \n-- \nShawn.\n"}]}