{"thread":{"id":"1103","subject":"[PATCH 0/2] Support for transferring pack files in git-ssh-*","startedAt":"2005-07-04T18:48:33Z","lastAt":"2005-07-04T18:51:39Z","messageCount":3,"participants":["Daniel Barkalow"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"5636","messageId":"Pine.LNX.4.21.0507041416240.30848-100000@iabervon.org","threadId":"1103","inReplyTo":null,"subject":"[PATCH 0/2] Support for transferring pack files in git-ssh-*","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-04T18:48:33Z","receivedAt":"2005-07-04T18:48:33Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"This series allows git-ssh-* to transfer objects packed into pack files in\nthe case of updating a ref file. It is a proof-of-concept for transferring\npack files in any situation where it's useful.\n\nThe general method is that the fetch() method has the option of\nadditionally getting other objects in addition to the one\nspecified; objects which aren't needed are specified with\ndont_fetch() (when it makes sense to exclude them). In this version, it\nonly excludes an object when it is the current value of a ref file that is\nbeing updated, but further exclusions are clearly possible.\n\nIn the case of git-ssh-*, the target specifies objects to exclude, and the\nsource responds (asynchronously) with whether or not it knows how to\nexclude them (i.e., whether or not it has them). If the target has gotten\nan object excluded, it requests a pack file instead of a single object,\nand the source provides all objects referenced from the given hash,\nexcluding those specified for exclusion.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"5637","messageId":"Pine.LNX.4.21.0507041448400.30848-100000@iabervon.org","threadId":"1103","inReplyTo":"Pine.LNX.4.21.0507041416240.30848-100000@iabervon.org","subject":"[PATCH 1/2] Specify object not useful to pull","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-04T18:50:31Z","receivedAt":"2005-07-04T18:50:31Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"Add support for the pull common code to specify to a pull implementation\nhashes which wouldn't be useful to fetch implicitly. This can be used to\ninfer (possibly) what hashes would be useful to fetch implicitly, such that\na later call to fetch can also fetch extra stuff.\n\nSigned-off-by: Daniel Barkalow <barkalow@iabervon.org>\n\n---\ncommit 9bc0256d2e834e101d7bd3f4867a330c92104929\ntree a2b4cfdaac954c46babffd585a61f3faf68969e1\nparent d0efc8a71da1855c705fd2074b219bcb158b6dbd\nauthor Daniel Barkalow <barkalow@iabervon.org> 1120411439 -0400\ncommitter Daniel Barkalow <barkalow@silva-tulga.(none)> 1120411439 -0400\n\nIndex: http-pull.c\n===================================================================\n--- f26b700095ec30154fede14638a099f49744981d/http-pull.c  (mode:100644 sha1:1f9d60b9b1d5eed85b24d96c240666bbfc5a22ed)\n+++ a2b4cfdaac954c46babffd585a61f3faf68969e1/http-pull.c  (mode:100644 sha1:f252a4b9d5448afa7af8a62176808a631429b9cd)\n@@ -139,6 +139,10 @@\n         return 0;\n }\n \n+void dont_fetch(const unsigned char *sha1)\n+{\n+}\n+\n int main(int argc, char **argv)\n {\n \tchar *commit_id;\nIndex: local-pull.c\n===================================================================\n--- f26b700095ec30154fede14638a099f49744981d/local-pull.c  (mode:100644 sha1:2f06fbee8b840a7ae642f5a22e2cb993687f3470)\n+++ a2b4cfdaac954c46babffd585a61f3faf68969e1/local-pull.c  (mode:100644 sha1:270e3a0b8405793cd70e6efa70ec6aa4b1674141)\n@@ -73,6 +73,10 @@\n \treturn -1;\n }\n \n+void dont_fetch(const unsigned char *sha1)\n+{\n+}\n+\n int fetch_ref(char *ref, unsigned char *sha1)\n {\n \tstatic int ref_name_start = -1;\nIndex: pull.c\n===================================================================\n--- f26b700095ec30154fede14638a099f49744981d/pull.c  (mode:100644 sha1:ed3078e3b27c62c07558fd94f339801cbd685593)\n+++ a2b4cfdaac954c46babffd585a61f3faf68969e1/pull.c  (mode:100644 sha1:f7f5a89aef36ffc2436dbd30170e4c8dbb2ba3a3)\n@@ -155,6 +155,10 @@\n \tunsigned char sha1[20];\n \tint fd = -1;\n \n+\tif (current_ref) {\n+\t\tdont_fetch(current_ref);\n+\t}\n+\n \tif (write_ref && current_ref) {\n \t\tfd = lock_ref_sha1(write_ref, current_ref);\n \t\tif (fd < 0)\nIndex: pull.h\n===================================================================\n--- f26b700095ec30154fede14638a099f49744981d/pull.h  (mode:100644 sha1:e173ae3337c4465da87d849f4e5c9da203fdf01d)\n+++ a2b4cfdaac954c46babffd585a61f3faf68969e1/pull.h  (mode:100644 sha1:6a35d39fd69bb884faa2d5e70c79e5c40b3ba436)\n@@ -15,6 +15,12 @@\n  */\n extern int fetch_ref(char *ref, unsigned char *sha1);\n \n+/*\n+ * Specify that the given SHA1, and everything it references, need not\n+ * be fetched.  To be provided by the particular implementation. \n+ */\n+extern void dont_fetch(const unsigned char *sha1);\n+\n /* If set, the ref filename to write the target value to. */\n extern const char *write_ref;\n \nIndex: ssh-pull.c\n===================================================================\n--- f26b700095ec30154fede14638a099f49744981d/ssh-pull.c  (mode:100644 sha1:87d523899a83d8c0d3c5ff721208ded30c1a38f4)\n+++ a2b4cfdaac954c46babffd585a61f3faf68969e1/ssh-pull.c  (mode:100644 sha1:362318071333420a7cf2450ada7269a94ec2cc7c)\n@@ -53,6 +53,10 @@\n \treturn 0;\n }\n \n+void dont_fetch(const unsigned char *sha1)\n+{\n+}\n+\n int main(int argc, char **argv)\n {\n \tchar *commit_id;\n"},{"id":"5638","messageId":"Pine.LNX.4.21.0507041450350.30848-100000@iabervon.org","threadId":"1103","inReplyTo":"Pine.LNX.4.21.0507041416240.30848-100000@iabervon.org","subject":"[PATCH 2/2] Pull well-bounded pack files over ssh","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-04T18:51:39Z","receivedAt":"2005-07-04T18:51:39Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"Initial version of support for transferring pack files over ssh. This works\nby sending the source hashes that the target already has, and checking (in\nbulk) whether the source has them as well. If the source has some hash that\nthe target has, the target will request a pack for the fetch, instead of\nrequesting individual objects, and the source will provide a pack that has\nthe requested object and all references, excluding those mentioned previously\nand any they refer to.\n\nSigned-off-by: Daniel Barkalow <barkalow@iabervon.org>\n\n---\ncommit 2045e6098dfa0f8760b6a4a65227a6ea51de990d\ntree 3d3e671c0ac57ea66434768c2b3432352b7c20ae\nparent a3d68b56cd5efaa06a7493aca61febb4f376fb49\nauthor Daniel Barkalow <barkalow@iabervon.org> 1120501001 -0400\ncommitter Daniel Barkalow <barkalow@silva-tulga.(none)> 1120501001 -0400\n\nIndex: ssh-pull.c\n===================================================================\n--- 8c3f58421885a9c1472f141bb32b668c2e65a5fc/ssh-pull.c  (mode:100644 sha1:0f2c3b5b9db324e20eb6ed0c52cc884b86349a08)\n+++ 3d3e671c0ac57ea66434768c2b3432352b7c20ae/ssh-pull.c  (mode:100644 sha1:f792135e7afba6a8d68946a7f14aa339c4fc02e7)\n@@ -4,11 +4,86 @@\n #include \"pull.h\"\n #include \"refs.h\"\n \n+#include \"pkt-line.h\"\n+#include <sys/wait.h>\n+\n+static const char *unpacker = \"git-unpack-objects\";\n+\n static int fd_in;\n static int fd_out;\n \n static unsigned char remote_version = 0;\n-static unsigned char local_version = 1;\n+static unsigned char local_version = 2;\n+\n+static int sent_excludes = 0;\n+static int accepted_excludes = 0;\n+\n+static int write_pack_from_fd(int fd)\n+{\n+\tpid_t pid = fork();\n+\n+\tif (pid < 0)\n+\t\tdie(\"unpack fork failed\");\n+\tif (!pid) {\n+\t\texeclp(unpacker, unpacker, NULL);\n+\t\tdie(\"unpack execute failed\");\n+\t}\n+\n+\tfor (;;) {\n+\t\tint status, code;\n+\t\tint retval = waitpid(pid, &status, 0);\n+\n+\t\tif (retval < 0) {\n+\t\t\tif (errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\tdie(\"waitpid failed (%s)\", strerror(retval));\n+\t\t}\n+\t\tif (retval != pid)\n+\t\t\tdie(\"waitpid is confused\");\n+\t\tif (WIFSIGNALED(status))\n+\t\t\tdie(\"%s died of signal %d\", unpacker, WTERMSIG(status));\n+\t\tif (!WIFEXITED(status))\n+\t\t\tdie(\"%s died out of really strange complications\", unpacker);\n+\t\tcode = WEXITSTATUS(status);\n+\t\tif (code)\n+\t\t\tdie(\"%s exited with error code %d\", unpacker, code);\n+\t\treturn 0;\n+\t}\n+}\n+\n+static int handle_deferred_reads()\n+{\n+\tsigned char remote;\n+\twhile (sent_excludes) {\n+\t\tif (read(fd_in, &remote, 1) < 1)\n+\t\t\treturn -1;\n+\t\tif (!remote) {\n+\t\t\tif (get_verbosely) {\n+\t\t\t\tfprintf(stderr, \"Exclude accepted\\n\");\n+\t\t\t}\n+\t\t\taccepted_excludes++;\n+\t\t}\n+\t\tsent_excludes--;\n+\t}\n+\treturn 0;\n+}\n+\n+static int fetch_pack(unsigned char *sha1)\n+{\n+\tint ret;\n+\tchar type = 'p';\n+\tsigned char remote;\n+\twrite(fd_out, &type, 1);\n+\twrite(fd_out, sha1, 20);\n+\tif (read(fd_in, &remote, 1) < 1)\n+\t\treturn -1;\n+\tif (remote < 0)\n+\t\treturn remote;\n+\tif (get_verbosely)\n+\t\tfprintf(stderr, \"Getting pack\\n\");\n+\tret = write_pack_from_fd(fd_in);\n+\treturn ret;\n+}\n \n int fetch(unsigned char *sha1)\n {\n@@ -17,6 +92,10 @@\n \tchar type = 'o';\n \tif (has_sha1_file(sha1))\n \t\treturn 0;\n+\tif (handle_deferred_reads())\n+\t\treturn -1;\n+\tif (accepted_excludes)\n+\t\treturn fetch_pack(sha1);\n \twrite(fd_out, &type, 1);\n \twrite(fd_out, sha1, 20);\n \tif (read(fd_in, &remote, 1) < 1)\n@@ -44,6 +123,8 @@\n {\n \tsigned char remote;\n \tchar type = 'r';\n+\tif (handle_deferred_reads())\n+\t\treturn -1;\n \twrite(fd_out, &type, 1);\n \twrite(fd_out, ref, strlen(ref) + 1);\n \tread(fd_in, &remote, 1);\n@@ -55,6 +136,15 @@\n \n void dont_fetch(const unsigned char *sha1)\n {\n+\tchar type = 'd';\n+\tif (remote_version < 2)\n+\t\treturn;\n+\twrite(fd_out, &type, 1);\n+\twrite(fd_out, sha1, 20);\n+\tsent_excludes++;\n+\tif (get_verbosely) {\n+\t\tfprintf(stderr, \"Won't get %s\\n\", sha1_to_hex(sha1));\n+\t}\n }\n \n int main(int argc, char **argv)\n@@ -76,7 +166,12 @@\n \t\t} else if (argv[arg][1] == 'v') {\n \t\t\tget_verbosely = 1;\n \t\t} else if (argv[arg][1] == 'w') {\n+\t\t\tunsigned char *current = malloc(20);\n \t\t\twrite_ref = argv[arg + 1];\n+\t\t\tif (get_ref_sha1(write_ref, current))\n+\t\t\t\tfree(current);\n+\t\t\telse\n+\t\t\t\tcurrent_ref = current;\n \t\t\targ++;\n \t\t}\n \t\targ++;\nIndex: ssh-push.c\n===================================================================\n--- 8c3f58421885a9c1472f141bb32b668c2e65a5fc/ssh-push.c  (mode:100644 sha1:090d6f9f8fbde2d736ac5bf563415b0fa402b5aa)\n+++ 3d3e671c0ac57ea66434768c2b3432352b7c20ae/ssh-push.c  (mode:100644 sha1:10390948efacfa06f4f6fc6b2f3631cec6fcb876)\n@@ -4,16 +4,88 @@\n \n #include <string.h>\n \n-static unsigned char local_version = 1;\n+static unsigned char local_version = 2;\n static unsigned char remote_version = 0;\n \n-static int serve_object(int fd_in, int fd_out) {\n+static void exec_pack_objects(void)\n+{\n+\tstatic char *args[] = {\n+\t\t\"git-pack-objects\",\n+\t\t\"--stdout\",\n+\t\tNULL\n+\t};\n+\texecvp(\"git-pack-objects\", args);\n+\tdie(\"git-pack-objects exec failed (%s)\", strerror(errno));\n+}\n+\n+static void exec_rev_list(unsigned char *incl, unsigned char *excl)\n+{\n+\tstatic char *args[1000];\n+\tchar *buf = malloc(100);\n+\tint i = 0;\n+\n+\targs[i++] = \"git-rev-list\";\t/* 0 */\n+\targs[i++] = \"--objects\";\t/* 1 */\n+\n+\targs[i++] = buf;\n+\tsnprintf(buf, 50, \"^%s\", sha1_to_hex(excl));\n+\tbuf += 50;\n+\targs[i++] = buf;\n+\tsnprintf(buf, 50, \"%s\", sha1_to_hex(incl));\n+\n+\targs[i] = NULL;\n+\texecvp(\"git-rev-list\", args);\n+\tdie(\"git-rev-list exec failed (%s)\", strerror(errno));\n+}\n+\n+static void rev_list(int fd, unsigned char *incl, unsigned char *excl)\n+{\n+\tint pipe_fd[2];\n+\tpid_t pack_objects_pid;\n+\n+\tif (pipe(pipe_fd) < 0)\n+\t\tdie(\"rev-list setup: pipe failed\");\n+\tpack_objects_pid = fork();\n+\tif (!pack_objects_pid) {\n+\t\tdup2(pipe_fd[0], 0);\n+\t\tdup2(fd, 1);\n+\t\tclose(pipe_fd[0]);\n+\t\tclose(pipe_fd[1]);\n+\t\tclose(fd);\n+\t\texec_pack_objects();\n+\t\tdie(\"pack-objects setup failed\");\n+\t}\n+\tif (pack_objects_pid < 0)\n+\t\tdie(\"pack-objects fork failed\");\n+\tdup2(pipe_fd[1], 1);\n+\tclose(pipe_fd[0]);\n+\tclose(pipe_fd[1]);\n+\tclose(fd);\n+\texec_rev_list(incl, excl);\n+}\n+\n+static int pack_objects(int fd, unsigned char *incl, unsigned char *excl)\n+{\n+\tpid_t rev_list_pid;\n+\n+\trev_list_pid = fork();\n+\tif (!rev_list_pid) {\n+\t\trev_list(fd, incl, excl);\n+\t\tdie(\"rev-list setup failed\");\n+\t}\n+\tif (rev_list_pid < 0)\n+\t\tdie(\"rev-list fork failed\");\n+\t/*\n+\t * We don't wait for the rev-list pipeline in the parent:\n+\t * we end up waiting for the other end instead\n+\t */\n+\treturn 0;\n+}\n+\n+static int read_sha1(int fd_in, unsigned char *sha1)\n+{\n \tssize_t size;\n \tint posn = 0;\n-\tunsigned char sha1[20];\n-\tunsigned long objsize;\n-\tvoid *buf;\n-\tsigned char remote;\n \tdo {\n \t\tsize = read(fd_in, sha1 + posn, 20 - posn);\n \t\tif (size < 0) {\n@@ -24,6 +96,18 @@\n \t\t\treturn -1;\n \t\tposn += size;\n \t} while (posn < 20);\n+\treturn 0;\n+}\n+\n+static int serve_object(int fd_in, int fd_out) {\n+\tunsigned char sha1[20];\n+\tunsigned long objsize;\n+\tvoid *buf;\n+\tsigned char remote;\n+\tint posn;\n+\tssize_t size;\n+\tif (read_sha1(fd_in, sha1))\n+\t\treturn -1;\n \t\n \t/* fprintf(stderr, \"Serving %s\\n\", sha1_to_hex(sha1)); */\n \tremote = 0;\n@@ -85,6 +169,48 @@\n         return 0;\n }\n \n+static unsigned char *excluded = NULL;\n+\n+static int serve_exclude(int fd_in, int fd_out)\n+{\n+\tunsigned char sha1[20];\n+\tsigned char remote;\n+\tif (read_sha1(fd_in, sha1))\n+\t\treturn -1;\n+\n+\tif (has_sha1_file(sha1)) {\n+\t\tremote = 0;\n+\t\tif (!excluded) {\n+\t\t\texcluded = malloc(20);\n+\t\t\tmemcpy(excluded, sha1, 20);\n+\t\t}\n+\t} else {\n+\t\tremote = -1;\n+\t}\n+\n+\twrite(fd_out, &remote, 1);\n+\t\n+\treturn 0;\n+}\n+\n+static int serve_pack(int fd_in, int fd_out)\n+{\n+\tunsigned char sha1[20];\n+\tsigned char remote = 0;\n+\tif (read_sha1(fd_in, sha1))\n+\t\treturn -1;\n+\n+\tif (!has_sha1_file(sha1))\n+\t\tremote = -1;\n+\n+\twrite(fd_out, &remote, 1);\n+\n+\tif (remote)\n+\t\treturn 0;\n+\tif (pack_objects(fd_out, sha1, excluded))\n+\t\treturn -1;\n+\treturn 0;\n+}\n \n static void service(int fd_in, int fd_out) {\n \tchar type;\n@@ -102,6 +228,10 @@\n \t\t\treturn;\n \t\tif (type == 'r' && serve_ref(fd_in, fd_out))\n \t\t\treturn;\n+\t\tif (type == 'd' && serve_exclude(fd_in, fd_out))\n+\t\t\treturn;\n+\t\tif (type == 'p' && serve_pack(fd_in, fd_out))\n+\t\t\treturn;\n \t} while (1);\n }\n \n"}]}