{"thread":{"id":"2188","subject":"[PATCH] Allow caching of generated pack for full cloning.","startedAt":"2005-10-22T09:00:12Z","lastAt":"2005-10-22T09:00:12Z","messageCount":1,"participants":["Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"10460","messageId":"7vhdbakjtv.fsf@assigned-by-dhcp.cox.net","threadId":"2188","inReplyTo":null,"subject":"[PATCH] Allow caching of generated pack for full cloning.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-22T09:00:12Z","receivedAt":"2005-10-22T09:00:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"git-pack-objects can reuse pack files stored in $GIT_DIR/pack-cache\ndirectory, when a necessary pack is found.  This is hopefully useful\nwhen upload-pack (called from git-daemon) is expected to receive\nrequests for the same set of objects many times (e.g full cloning\nrequest of any project, or updates from the set of heads previous day\nto the latest for a slow moving project).\n\nCurrently git-pack-objects does *not* keep pack files it creates for\nreusing.  It might be useful to implement its --update-cache option,\nwhich would let it store pack files it created in the pack-cache\ndirectory, and prune rarely used ones from it.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n\n---\n\n  Right now, this is not very useful except perhaps preparing\n  for many clone requests by statically creating and storing a\n  full pack in pack-cache directory.  I am expecting that\n  enabling the unimplemented --update-cache option of\n  git-pack-objects would let the server keep recently generated\n  packs, hoping that fetch requests close together would be for\n  the same \"master\" head, relative to the same previous heads\n  (multiple people making a habit of pulling every day, or every\n  week, or whatever).  These cached packs need to be purged from\n  the pack-cache directory quite often.  They would become\n  useless once you update a popular ref in the repository.\n\n  Even if this caching would help git-daemon by reusing\n  generated packs, I see one potential problem; --update-cache\n  option would require the process to be able to write into the\n  pack-cache directory, but I expect git-daemon would run as a\n  user that does not have any write privilege to the filesystem.\n\n Makefile       |    2 +\n cache.h        |    1 +\n copy.c         |   37 +++++++++++++++++++++++\n pack-objects.c |   90 +++++++++++++++++++++++++++++++++++++++++++++++++-------\n upload-pack.c  |   12 ++++++-\n 5 files changed, 126 insertions(+), 16 deletions(-)\n create mode 100644 copy.c\n\napplies-to: a0d57ba5b9245eb3a4cc15fb029af51a40eb8136\ndd42e422104f43b369929c4f900362d401d2e962\ndiff --git a/Makefile b/Makefile\nindex 903c57c..3d8503d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -159,7 +159,7 @@ LIB_OBJS = \\\n \tobject.o pack-check.o patch-delta.o path.o pkt-line.o \\\n \tquote.o read-cache.o refs.o run-command.o \\\n \tserver-info.o setup.o sha1_file.o sha1_name.o strbuf.o \\\n-\ttag.o tree.o usage.o config.o environment.o ctype.o \\\n+\ttag.o tree.o usage.o config.o environment.o ctype.o copy.o \\\n \t$(DIFF_OBJS)\n \n LIBS = $(LIB_FILE)\ndiff --git a/cache.h b/cache.h\nindex d776016..2e36cc5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -413,4 +413,5 @@ static inline int sane_case(int x, int h\n \treturn x;\n }\n \n+extern int copy_fd(int ifd, int ofd);\n #endif /* CACHE_H */\ndiff --git a/copy.c b/copy.c\nnew file mode 100644\nindex 0000000..2009275\n--- /dev/null\n+++ b/copy.c\n@@ -0,0 +1,37 @@\n+#include \"cache.h\"\n+\n+int copy_fd(int ifd, int ofd)\n+{\n+\twhile (1) {\n+\t\tint len;\n+\t\tchar buffer[8192];\n+\t\tchar *buf = buffer;\n+\t\tlen = read(ifd, buffer, sizeof(buffer));\n+\t\tif (!len)\n+\t\t\tbreak;\n+\t\tif (len < 0) {\n+\t\t\tif (errno == EAGAIN)\n+\t\t\t\tcontinue;\n+\t\t\treturn error(\"copy-fd: read returned %s\",\n+\t\t\t\t     strerror(errno));\n+\t\t}\n+\t\twhile (1) {\n+\t\t\tint written = write(ofd, buf, len);\n+\t\t\tif (written > 0) {\n+\t\t\t\tbuf += written;\n+\t\t\t\tlen -= written;\n+\t\t\t\tif (!len)\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tif (!written)\n+\t\t\t\treturn error(\"copy-fd: write returned 0\");\n+\t\t\tif (errno == EAGAIN || errno == EINTR)\n+\t\t\t\tcontinue;\n+\t\t\treturn error(\"copy-fd: write returned %s\",\n+\t\t\t\t     strerror(errno));\n+\t\t}\n+\t}\n+\tclose(ifd);\n+\treturn 0;\n+}\n+\ndiff --git a/pack-objects.c b/pack-objects.c\nindex b3e6152..915469e 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -4,7 +4,7 @@\n #include \"pack.h\"\n #include \"csum-file.h\"\n \n-static const char pack_usage[] = \"git-pack-objects [--local] [--incremental] [--window=N] [--depth=N] {--stdout | base-name} < object-list\";\n+static const char pack_usage[] = \"git-pack-objects [--local] [--incremental] [--window=N] [--depth=N] {--stdout | base-name} [--update-cache] < object-list\";\n \n struct object_entry {\n \tunsigned char sha1[20];\n@@ -400,6 +400,71 @@ static void find_deltas(struct object_en\n \tfree(array);\n }\n \n+static void prepare_pack(int window, int depth)\n+{\n+\tget_object_details();\n+\n+\tfprintf(stderr, \"Packing %d objects\\n\", nr_objects);\n+\n+\tsorted_by_type = create_sorted_list(type_size_sort);\n+\tif (window && depth)\n+\t\tfind_deltas(sorted_by_type, window+1, depth);\n+\twrite_pack_file();\n+}\n+\n+static int reuse_cached_pack(unsigned char *sha1, int pack_to_stdout)\n+{\n+\tstatic const char cache[] = \"pack-cache/pack-%s.%s\";\n+\tchar *cached_pack, *cached_idx;\n+\tint ifd, ofd, ifd_ix = -1;\n+\n+\tcached_pack = git_path(cache, sha1_to_hex(sha1), \"pack\");\n+\tifd = open(cached_pack, O_RDONLY);\n+\tif (ifd < 0)\n+\t\treturn 0;\n+\n+\tif (!pack_to_stdout) {\n+\t\tcached_idx = git_path(cache, sha1_to_hex(sha1), \"idx\");\n+\t\tifd_ix = open(cached_idx, O_RDONLY);\n+\t\tif (ifd_ix < 0) {\n+\t\t\tclose(ifd);\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\tfprintf(stderr, \"Reusing %d objects pack %s\\n\", nr_objects,\n+\t\tsha1_to_hex(sha1));\n+\n+\tif (pack_to_stdout) {\n+\t\tif (copy_fd(ifd, 1))\n+\t\t\texit(1);\n+\t\tclose(ifd);\n+\t}\n+\telse {\n+\t\tchar name[PATH_MAX];\n+\t\tsnprintf(name, sizeof(name),\n+\t\t\t \"%s-%s.%s\", base_name, sha1_to_hex(sha1), \"pack\");\n+\t\tofd = open(name, O_CREAT | O_EXCL | O_WRONLY, 0666);\n+\t\tif (ofd < 0)\n+\t\t\tdie(\"unable to open %s (%s)\", name, strerror(errno));\n+\t\tif (copy_fd(ifd, ofd))\n+\t\t\texit(1);\n+\t\tclose(ifd);\n+\n+\t\tsnprintf(name, sizeof(name),\n+\t\t\t \"%s-%s.%s\", base_name, sha1_to_hex(sha1), \"idx\");\n+\t\tofd = open(name, O_CREAT | O_EXCL | O_WRONLY, 0666);\n+\t\tif (ofd < 0)\n+\t\t\tdie(\"unable to open %s (%s)\", name, strerror(errno));\n+\t\tif (copy_fd(ifd_ix, ofd))\n+\t\t\texit(1);\n+\t\tclose(ifd_ix);\n+\t\tputs(sha1_to_hex(sha1));\n+\t}\n+\n+\treturn 1;\n+}\n+\n int main(int argc, char **argv)\n {\n \tSHA_CTX ctx;\n@@ -424,6 +489,10 @@ int main(int argc, char **argv)\n \t\t\t\tincremental = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(\"--update-cache\", arg)) {\n+\t\t\t\t/* Not implemented */\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strncmp(\"--window=\", arg, 9)) {\n \t\t\t\tchar *end;\n \t\t\t\twindow = strtoul(arg+9, &end, 0);\n@@ -472,9 +541,6 @@ int main(int argc, char **argv)\n \t}\n \tif (non_empty && !nr_objects)\n \t\treturn 0;\n-\tget_object_details();\n-\n-\tfprintf(stderr, \"Packing %d objects\\n\", nr_objects);\n \n \tsorted_by_sha = create_sorted_list(sha1_sort);\n \tSHA1_Init(&ctx);\n@@ -485,14 +551,14 @@ int main(int argc, char **argv)\n \t}\n \tSHA1_Final(object_list_sha1, &ctx);\n \n-\tsorted_by_type = create_sorted_list(type_size_sort);\n-\tif (window && depth)\n-\t\tfind_deltas(sorted_by_type, window+1, depth);\n-\n-\twrite_pack_file();\n-\tif (!pack_to_stdout) {\n-\t\twrite_index_file();\n-\t\tputs(sha1_to_hex(object_list_sha1));\n+\tif (reuse_cached_pack(object_list_sha1, pack_to_stdout))\n+\t\t;\n+\telse {\n+\t\tprepare_pack(window, depth);\n+\t\tif (!pack_to_stdout) {\n+\t\t\twrite_index_file();\n+\t\t\tputs(sha1_to_hex(object_list_sha1));\n+\t\t}\n \t}\n \treturn 0;\n }\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 8a41caf..6fb8eb7 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -29,6 +29,7 @@ static void create_pack_file(void)\n {\n \tint fd[2];\n \tpid_t pid;\n+\tint create_full_pack = (MAX_NEEDS <= nr_needs);\n \n \tif (pipe(fd) < 0)\n \t\tdie(\"git-upload-pack: unable to create pipe\");\n@@ -43,7 +44,7 @@ static void create_pack_file(void)\n \t\tchar *buf;\n \t\tchar **p;\n \n-\t\tif (MAX_NEEDS <= nr_needs)\n+\t\tif (create_full_pack)\n \t\t\targs = nr_has + 10;\n \t\telse\n \t\t\targs = nr_has + nr_needs + 5;\n@@ -57,7 +58,7 @@ static void create_pack_file(void)\n \t\tclose(fd[1]);\n \t\t*p++ = \"git-rev-list\";\n \t\t*p++ = \"--objects\";\n-\t\tif (MAX_NEEDS <= nr_needs)\n+\t\tif (create_full_pack)\n \t\t\t*p++ = \"--all\";\n \t\telse {\n \t\t\tfor (i = 0; i < nr_needs; i++) {\n@@ -79,7 +80,12 @@ static void create_pack_file(void)\n \tdup2(fd[0], 0);\n \tclose(fd[0]);\n \tclose(fd[1]);\n-\texeclp(\"git-pack-objects\", \"git-pack-objects\", \"--stdout\", NULL);\n+\tif (create_full_pack)\n+\t\texeclp(\"git-pack-objects\", \"git-pack-objects\",\n+\t\t       \"--stdout\", NULL);\n+\telse\n+\t\texeclp(\"git-pack-objects\", \"git-pack-objects\",\n+\t\t       \"--stdout\", \"--update-cache\", NULL);\n \tdie(\"git-upload-pack: unable to exec git-pack-objects\");\n }\n \n---\n0.99.8.GIT\n"}]}