{"thread":{"id":"4602","subject":"[PATCH 4/4] upload-pack/fetch-pack: support side-band communication","startedAt":"2006-06-21T11:01:47Z","lastAt":"2006-06-22T16:22:14Z","messageCount":3,"participants":["Junio C Hamano","Jon Loeliger"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"22198","messageId":"7vsllyzs9w.fsf@assigned-by-dhcp.cox.net","threadId":"4602","inReplyTo":null,"subject":"[PATCH 4/4] upload-pack/fetch-pack: support side-band communication","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-06-21T11:01:47Z","receivedAt":"2006-06-21T11:01:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This implements a protocol extension between fetch-pack and\nupload-pack to allow stderr stream from upload-pack (primarily\nused for the progress bar display) to be passed back.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n---\n\n * With this, fetching and cloning over git+ssh:// and git://\n   does not discard the progress bar and error messages that\n   upload-pack emits on its standard output.  They are sent to\n   the downloader and shown to the user.\n\n   Somehow this does not work when connecting to git daemon that\n   runs under inetd.  Fixes appreciated.\n\n cache.h       |    4 ++-\n fetch-clone.c |   71 +++++++++++++++++++++++++++++++++++++++++++++++++++++----\n fetch-pack.c  |   22 +++++++++++++-----\n pkt-line.c    |    4 ++-\n pkt-line.h    |    1 +\n upload-pack.c |   60 +++++++++++++++++++++++++++++++++++++++++-------\n 6 files changed, 139 insertions(+), 23 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex eaa5c0c..efeafea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -374,8 +374,8 @@ extern char git_commit_encoding[MAX_ENCO\n extern int copy_fd(int ifd, int ofd);\n \n /* Finish off pack transfer receiving end */\n-extern int receive_unpack_pack(int fd[2], const char *me, int quiet);\n-extern int receive_keep_pack(int fd[2], const char *me, int quiet);\n+extern int receive_unpack_pack(int fd[2], const char *me, int quiet, int);\n+extern int receive_keep_pack(int fd[2], const char *me, int quiet, int);\n \n /* pager.c */\n extern void setup_pager(void);\ndiff --git a/fetch-clone.c b/fetch-clone.c\nindex da1b3ff..c16b0c4 100644\n--- a/fetch-clone.c\n+++ b/fetch-clone.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"exec_cmd.h\"\n+#include \"pkt-line.h\"\n #include <sys/wait.h>\n #include <sys/time.h>\n \n@@ -23,7 +24,7 @@ static int finish_pack(const char *pack_\n \n \tpid = fork();\n \tif (pid < 0)\n-\t\tdie(\"git-clone-pack: unable to fork off git-index-pack\");\n+\t\tdie(\"%s: unable to fork off git-index-pack\", me);\n \tif (!pid) {\n \t\tclose(0);\n \t\tdup2(pipe_fd[1], 1);\n@@ -94,11 +95,69 @@ static int finish_pack(const char *pack_\n \texit(1);\n }\n \n-int receive_unpack_pack(int fd[2], const char *me, int quiet)\n+static pid_t setup_sideband(int sideband, const char *me, int fd[2], int xd[2])\n+{\n+\tpid_t side_pid;\n+\n+\tif (!sideband) {\n+\t\tfd[0] = xd[0];\n+\t\tfd[1] = xd[1];\n+\t\treturn 0;\n+\t}\n+\t/* xd[] is talking with upload-pack; subprocess reads from\n+\t * xd[0], spits out band#2 to stderr, and feeds us band#1\n+\t * through our fd[0].\n+\t */\n+\tif (pipe(fd) < 0)\n+\t\tdie(\"%s: unable to set up pipe\", me);\n+\tside_pid = fork();\n+\tif (side_pid < 0)\n+\t\tdie(\"%s: unable to fork off sideband demultiplexer\", me);\n+\tif (!side_pid) {\n+\t\t/* subprocess */\n+\t\tclose(fd[0]);\n+\t\tif (xd[0] != xd[1])\n+\t\t\tclose(xd[1]);\n+\t\twhile (1) {\n+\t\t\tchar buf[1024];\n+\t\t\tint len = packet_read_line(xd[0], buf, sizeof(buf));\n+\t\t\tif (len == 0)\n+\t\t\t\tbreak;\n+\t\t\tif (len < 1)\n+\t\t\t\tdie(\"%s: protocol error: no band designator\",\n+\t\t\t\t    me);\n+\t\t\tlen--;\n+\t\t\tswitch (buf[0] & 0xFF) {\n+\t\t\tcase 3:\n+\t\t\t\tsafe_write(2, buf+1, len);\n+\t\t\t\tfprintf(stderr, \"\\n\");\n+\t\t\t\texit(1);\n+\t\t\tcase 2:\n+\t\t\t\tsafe_write(2, buf+1, len);\n+\t\t\t\tcontinue;\n+\t\t\tcase 1:\n+\t\t\t\tsafe_write(fd[1], buf+1, len);\n+\t\t\t\tcontinue;\n+\t\t\tdefault:\n+\t\t\t\tdie(\"%s: protocol error: bad band #%d\",\n+\t\t\t\t    me, (buf[0] & 0xFF));\n+\t\t\t}\n+\t\t}\n+\t\texit(0);\n+\t}\n+\tclose(xd[0]);\n+\tclose(fd[1]);\n+\tfd[1] = xd[1];\n+\treturn side_pid;\n+}\n+\n+int receive_unpack_pack(int xd[2], const char *me, int quiet, int sideband)\n {\n \tint status;\n-\tpid_t pid;\n+\tpid_t pid, side_pid;\n+\tint fd[2];\n \n+\tside_pid = setup_sideband(sideband, me, fd, xd);\n \tpid = fork();\n \tif (pid < 0)\n \t\tdie(\"%s: unable to fork off git-unpack-objects\", me);\n@@ -147,10 +206,10 @@ #define NR_AVERAGE (4)\n  */\n #define usec_to_binarymsec(x) ((int)(x) / (1000512 >> 10))\n \n-int receive_keep_pack(int fd[2], const char *me, int quiet)\n+int receive_keep_pack(int xd[2], const char *me, int quiet, int sideband)\n {\n \tchar tmpfile[PATH_MAX];\n-\tint ofd, ifd;\n+\tint ofd, ifd, fd[2];\n \tunsigned long total;\n \tstatic struct timeval prev_tv;\n \tstruct average {\n@@ -160,6 +219,8 @@ int receive_keep_pack(int fd[2], const c\n \tunsigned long avg_bytes, avg_time;\n \tint idx = 0;\n \n+\tsetup_sideband(sideband, me, fd, xd);\n+\n \tifd = fd[0];\n \tsnprintf(tmpfile, sizeof(tmpfile),\n \t\t \"%s/pack/tmp-XXXXXX\", get_object_directory());\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7d23a80..f2c51eb 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -25,7 +25,7 @@ #define POPPED\t\t(1U << 4)\n #define MAX_IN_VAIN 256\n \n static struct commit_list *rev_list = NULL;\n-static int non_common_revs = 0, multi_ack = 0, use_thin_pack = 0;\n+static int non_common_revs = 0, multi_ack = 0, use_thin_pack = 0, use_sideband;\n \n static void rev_list_push(struct commit *commit, int mark)\n {\n@@ -165,9 +165,14 @@ static int find_common(int fd[2], unsign\n \t\t\tcontinue;\n \t\t}\n \n-\t\tpacket_write(fd[1], \"want %s%s%s\\n\", sha1_to_hex(remote),\n-\t\t\t     (multi_ack ? \" multi_ack\" : \"\"),\n-\t\t\t     (use_thin_pack ? \" thin-pack\" : \"\"));\n+\t\tif (!fetching)\n+\t\t\tpacket_write(fd[1], \"want %s%s%s%s\\n\",\n+\t\t\t\t     sha1_to_hex(remote),\n+\t\t\t\t     (multi_ack ? \" multi_ack\" : \"\"),\n+\t\t\t\t     (use_sideband ? \" side-band\" : \"\"),\n+\t\t\t\t     (use_thin_pack ? \" thin-pack\" : \"\"));\n+\t\telse\n+\t\t\tpacket_write(fd[1], \"want %s\\n\", sha1_to_hex(remote));\n \t\tfetching++;\n \t}\n \tpacket_flush(fd[1]);\n@@ -421,6 +426,11 @@ static int fetch_pack(int fd[2], int nr_\n \t\t\tfprintf(stderr, \"Server supports multi_ack\\n\");\n \t\tmulti_ack = 1;\n \t}\n+\tif (server_supports(\"side-band\")) {\n+\t\tif (verbose)\n+\t\t\tfprintf(stderr, \"Server supports side-band\\n\");\n+\t\tuse_sideband = 1;\n+\t}\n \tif (!ref) {\n \t\tpacket_flush(fd[1]);\n \t\tdie(\"no matching remote head\");\n@@ -437,9 +447,9 @@ static int fetch_pack(int fd[2], int nr_\n \t\t\tfprintf(stderr, \"warning: no common commits\\n\");\n \n \tif (keep_pack)\n-\t\tstatus = receive_keep_pack(fd, \"git-fetch-pack\", quiet);\n+\t\tstatus = receive_keep_pack(fd, \"git-fetch-pack\", quiet, use_sideband);\n \telse\n-\t\tstatus = receive_unpack_pack(fd, \"git-fetch-pack\", quiet);\n+\t\tstatus = receive_unpack_pack(fd, \"git-fetch-pack\", quiet, use_sideband);\n \n \tif (status)\n \t\tdie(\"git-fetch-pack: fetch failed.\");\ndiff --git a/pkt-line.c b/pkt-line.c\nindex bb3bab0..3d724ac 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -16,8 +16,9 @@ #include \"pkt-line.h\"\n  * The writing side could use stdio, but since the reading\n  * side can't, we stay with pure read/write interfaces.\n  */\n-static void safe_write(int fd, const void *buf, unsigned n)\n+ssize_t safe_write(int fd, const void *buf, ssize_t n)\n {\n+\tssize_t nn = n;\n \twhile (n) {\n \t\tint ret = xwrite(fd, buf, n);\n \t\tif (ret > 0) {\n@@ -29,6 +30,7 @@ static void safe_write(int fd, const voi\n \t\t\tdie(\"write error (disk full?)\");\n \t\tdie(\"write error (%s)\", strerror(errno));\n \t}\n+\treturn nn;\n }\n \n /*\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 51d0cbe..9abef24 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -8,5 +8,6 @@ void packet_flush(int fd);\n void packet_write(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n \n int packet_read_line(int fd, char *buffer, unsigned size);\n+ssize_t safe_write(int, const void *, ssize_t);\n \n #endif\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 13eaa22..7b86f69 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -21,6 +21,7 @@ static int use_thin_pack = 0;\n static unsigned char has_sha1[MAX_HAS][20];\n static unsigned char needs_sha1[MAX_NEEDS][20];\n static unsigned int timeout = 0;\n+static int use_sideband = 0;\n \n static void reset_timeout(void)\n {\n@@ -34,6 +35,43 @@ static int strip(char *line, int len)\n \treturn len;\n }\n \n+#define PACKET_MAX 1000\n+static ssize_t send_client_data(int fd, const char *data, ssize_t sz)\n+{\n+\tssize_t ssz;\n+\tconst char *p;\n+\n+\tif (!data) {\n+\t\tif (!use_sideband)\n+\t\t\treturn 0;\n+\t\tpacket_flush(1);\n+\t}\n+\n+\tif (!use_sideband) {\n+\t\tif (fd == 3)\n+\t\t\t/* emergency quit */\n+\t\t\tfd = 2;\n+\t\treturn safe_write(fd, data, sz);\n+\t}\n+\tp = data;\n+\tssz = sz;\n+\twhile (sz) {\n+\t\tunsigned n;\n+\t\tchar hdr[5];\n+\n+\t\tn = sz;\n+\t\tif (PACKET_MAX - 5 < n)\n+\t\t\tn = PACKET_MAX - 5;\n+\t\tsprintf(hdr, \"%04x\", n + 5);\n+\t\thdr[4] = fd;\n+\t\tsafe_write(1, hdr, 5);\n+\t\tsafe_write(1, p, n);\n+\t\tp += n;\n+\t\tsz -= n;\n+\t}\n+\treturn ssz;\n+}\n+\n static void create_pack_file(void)\n {\n \t/* Pipes between rev-list to pack-objects, pack-objects to us\n@@ -43,6 +81,8 @@ static void create_pack_file(void)\n \tpid_t pid_rev_list, pid_pack_objects;\n \tint create_full_pack = (nr_our_refs == nr_needs && !nr_has);\n \tchar data[8193], progress[128];\n+\tchar abort_msg[] = \"aborting due to possible repository \"\n+\t\t\"corruption on the remote side.\";\n \tint buffered = -1;\n \n \tif (pipe(lp_pipe) < 0)\n@@ -132,7 +172,6 @@ static void create_pack_file(void)\n \n \twhile (1) {\n \t\tconst char *who;\n-\t\tchar *cp;\n \t\tstruct pollfd pfd[2];\n \t\tpid_t pid;\n \t\tint status;\n@@ -199,19 +238,18 @@ static void create_pack_file(void)\n \t\t\t\t}\n \t\t\t\telse\n \t\t\t\t\tbuffered = -1;\n-\t\t\t\tsz = xwrite(1, data, sz);\n+\t\t\t\tsz = send_client_data(1, data, sz);\n \t\t\t\tif (sz < 0)\n \t\t\t\t\tgoto fail;\n \t\t\t}\n \t\t\tif (0 <= pe && (pfd[pe].revents & (POLLIN|POLLHUP))) {\n-\t\t\t\t/* Status ready; we do not use it for now,\n-\t\t\t\t * but later we will add side-band to send it\n-\t\t\t\t * to the other side.\n+\t\t\t\t/* Status ready; we ship that in the side-band\n+\t\t\t\t * or dump to the standard error.\n \t\t\t\t */\n \t\t\t\tsz = read(pe_pipe[0], progress,\n \t\t\t\t\t  sizeof(progress));\n \t\t\t\tif (0 < sz)\n-\t\t\t\t\twrite(2, progress, sz);\n+\t\t\t\t\tsend_client_data(2, progress, sz);\n \t\t\t\telse if (sz == 0) {\n \t\t\t\t\tclose(pe_pipe[0]);\n \t\t\t\t\tpe_pipe[0] = -1;\n@@ -259,11 +297,12 @@ static void create_pack_file(void)\n \t\t/* flush the data */\n \t\tif (0 <= buffered) {\n \t\t\tdata[0] = buffered;\n-\t\t\tsz = xwrite(1, data, 1);\n+\t\t\tsz = send_client_data(1, data, 1);\n \t\t\tif (sz < 0)\n \t\t\t\tgoto fail;\n \t\t\tfprintf(stderr, \"flushed.\\n\");\n \t\t}\n+\t\tsend_client_data(1, NULL, 0);\n \t\treturn;\n \t}\n  fail:\n@@ -271,7 +310,8 @@ static void create_pack_file(void)\n \t\tkill(pid_pack_objects, SIGKILL);\n \tif (pid_rev_list)\n \t\tkill(pid_rev_list, SIGKILL);\n-\tdie(\"git-upload-pack: aborting due to possible repository corruption on the remote side.\");\n+\tsend_client_data(3, abort_msg, sizeof(abort_msg));\n+\tdie(\"git-upload-pack: %s\", abort_msg);\n }\n \n static int got_sha1(char *hex, unsigned char *sha1)\n@@ -378,6 +418,8 @@ static int receive_needs(void)\n \t\t\tmulti_ack = 1;\n \t\tif (strstr(line+45, \"thin-pack\"))\n \t\t\tuse_thin_pack = 1;\n+\t\tif (strstr(line+45, \"side-band\"))\n+\t\t\tuse_sideband = 1;\n \n \t\t/* We have sent all our refs already, and the other end\n \t\t * should have chosen out of them; otherwise they are\n@@ -399,7 +441,7 @@ static int receive_needs(void)\n \n static int send_ref(const char *refname, const unsigned char *sha1)\n {\n-\tstatic char *capabilities = \"multi_ack thin-pack\";\n+\tstatic char *capabilities = \"multi_ack thin-pack side-band\";\n \tstruct object *o = parse_object(sha1);\n \n \tif (!o)\n-- \n1.4.0.g160a\n"},{"id":"22227","messageId":"7vpsh2xcv1.fsf@assigned-by-dhcp.cox.net","threadId":"4602","inReplyTo":"7vsllyzs9w.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 4/4] upload-pack/fetch-pack: support side-band communication","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-06-22T00:17:38Z","receivedAt":"2006-06-22T00:17:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <junkio@cox.net> writes:\n\n> This implements a protocol extension between fetch-pack and\n> upload-pack to allow stderr stream from upload-pack (primarily\n> used for the progress bar display) to be passed back.\n>\n> Signed-off-by: Junio C Hamano <junkio@cox.net>\n> ---\n>\n>  * With this, fetching and cloning over git+ssh:// and git://\n>    does not discard the progress bar and error messages that\n>    upload-pack emits on its standard output.  They are sent to\n>    the downloader and shown to the user.\n>\n>    Somehow this does not work when connecting to git daemon that\n>    runs under inetd.  Fixes appreciated.\n\nThis seems to fix it.\n\n-- >8 --\nSubject: [PATCH] daemon: send stderr to /dev/null instead of closing.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n---\n daemon.c |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/daemon.c b/daemon.c\nindex bdfe80d..0747ce2 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -757,7 +757,7 @@ int main(int argc, char **argv)\n \t\tstruct sockaddr *peer = (struct sockaddr *)&ss;\n \t\tsocklen_t slen = sizeof(ss);\n \n-\t\tfclose(stderr); //FIXME: workaround\n+\t\tfreopen(\"/dev/null\", \"w\", stderr);\n \n \t\tif (getpeername(0, peer, &slen))\n \t\t\tpeer = NULL;\n-- \n1.4.0.g7883\n"},{"id":"22282","messageId":"1150993333.9022.51.camel@cashmere.sps.mot.com","threadId":"4602","inReplyTo":"7vpsh2xcv1.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 4/4] upload-pack/fetch-pack: support side-band communication","fromName":"Jon Loeliger","fromEmail":"jdl@freescale.com","sentAt":"2006-06-22T16:22:14Z","receivedAt":"2006-06-22T16:22:14Z","isPatch":true,"sender":{"key":"jdl@jdl.com","avatar":"https://gravatar.com/avatar/75ce9a10b151acd2c28ec4ab2136dba7b2ff1634530bd04b155981a749d08a64?d=mp&s=160"},"body":"On Wed, 2006-06-21 at 19:17, Junio C Hamano wrote:\n> >    Somehow this does not work when connecting to git daemon that\n> >    runs under inetd.  Fixes appreciated.\n> \n> This seems to fix it.\n\n> -\t\tfclose(stderr); //FIXME: workaround\n> +\t\tfreopen(\"/dev/null\", \"w\", stderr);\n>  \n\nSweet.  Thanks!\n\njdl\n"}]}