{"thread":{"id":"1380","subject":"[PATCH 0/2] Support pack files in http-pull","startedAt":"2005-07-31T20:09:36Z","lastAt":"2005-08-01T08:16:39Z","messageCount":7,"participants":["barkalow@iabervon.org","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"6673","messageId":"Pine.LNX.4.62.0507311600040.23721@iabervon.org","threadId":"1380","inReplyTo":null,"subject":"[PATCH 0/2] Support pack files in http-pull","fromName":"","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-31T20:09:36Z","receivedAt":"2005-07-31T20:09:36Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"This series adds support for downloading pack files when appropriate in \nhttp-pull. When it finds that a needed object is not available, it \ndownloads info/packs (into memory), identifies any pack files it doesn't \nhave from there, downloads indices of any of these that it doesn't have, \nand downloads the pack containing the object. If other packs are also \nneeded, it downloads them when it reaches them.\n\n \t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"6675","messageId":"Pine.LNX.4.62.0507311609420.23721@iabervon.org","threadId":"1380","inReplyTo":"Pine.LNX.4.62.0507311600040.23721@iabervon.org","subject":"[PATCH 1/2] Functions for managing the set of packs the library is using","fromName":"","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-31T20:10:22Z","receivedAt":"2005-07-31T20:10:22Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"This adds support for reading an uninstalled index, and installing a\npack file that was added while the program was running, as well as\nfunctions for determining where to put the file.\n\nSigned-off-by: Daniel Barkalow <barkalow@iabervon.org>\n---\n\n  cache.h     |   13 ++++++\n  sha1_file.c |  123 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n  2 files changed, 135 insertions(+), 1 deletions(-)\n\n20fcc8f66a6780cf9bbd2fc2ba3b918c33696a67\ndiff --git a/cache.h b/cache.h\n--- a/cache.h\n+++ b/cache.h\n@@ -172,6 +172,8 @@ extern void rollback_index_file(struct c\n  extern char *mkpath(const char *fmt, ...);\n  extern char *git_path(const char *fmt, ...);\n  extern char *sha1_file_name(const unsigned char *sha1);\n+extern char *sha1_pack_name(const unsigned char *sha1);\n+extern char *sha1_pack_index_name(const unsigned char *sha1);\n\n  int safe_create_leading_directories(char *path);\n\n@@ -200,6 +202,9 @@ extern int write_sha1_to_fd(int fd, cons\n  extern int has_sha1_pack(const unsigned char *sha1);\n  extern int has_sha1_file(const unsigned char *sha1);\n\n+extern int has_pack_file(const unsigned char *sha1);\n+extern int has_pack_index(const unsigned char *sha1);\n+\n  /* Convert to/from hex/sha1 representation */\n  extern int get_sha1(const char *str, unsigned char *sha1);\n  extern int get_sha1_hex(const char *hex, unsigned char *sha1);\n@@ -276,6 +281,7 @@ extern struct packed_git {\n  \tvoid *pack_base;\n  \tunsigned int pack_last_used;\n  \tunsigned int pack_use_cnt;\n+\tunsigned char sha1[20];\n  \tchar pack_name[0]; /* something like \".git/objects/pack/xxxxx.pack\" */\n  } *packed_git;\n\n@@ -298,7 +304,14 @@ extern int path_match(const char *path,\n  extern int get_ack(int fd, unsigned char *result_sha1);\n  extern struct ref **get_remote_heads(int in, struct ref **list, int nr_match, char **match);\n\n+extern struct packed_git *parse_pack_index(unsigned char *sha1);\n+\n  extern void prepare_packed_git(void);\n+extern void install_packed_git(struct packed_git *pack);\n+\n+extern struct packed_git *find_sha1_pack(const unsigned char *sha1, \n+\t\t\t\t\t struct packed_git *packs);\n+\n  extern int use_packed_git(struct packed_git *);\n  extern void unuse_packed_git(struct packed_git *);\n  extern struct packed_git *add_packed_git(char *, int);\ndiff --git a/sha1_file.c b/sha1_file.c\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -200,6 +200,56 @@ char *sha1_file_name(const unsigned char\n  \treturn base;\n  }\n\n+char *sha1_pack_name(const unsigned char *sha1)\n+{\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\tstatic char *name, *base, *buf;\n+\tint i;\n+\n+\tif (!base) {\n+\t\tconst char *sha1_file_directory = get_object_directory();\n+\t\tint len = strlen(sha1_file_directory);\n+\t\tbase = xmalloc(len + 60);\n+\t\tsprintf(base, \"%s/pack/pack-1234567890123456789012345678901234567890.pack\", sha1_file_directory);\n+\t\tname = base + len + 11;\n+\t}\n+\n+\tbuf = name;\n+\n+\tfor (i = 0; i < 20; i++) {\n+\t\tunsigned int val = *sha1++;\n+\t\t*buf++ = hex[val >> 4];\n+\t\t*buf++ = hex[val & 0xf];\n+\t}\n+ \n+\treturn base;\n+}\n+\n+char *sha1_pack_index_name(const unsigned char *sha1)\n+{\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\tstatic char *name, *base, *buf;\n+\tint i;\n+\n+\tif (!base) {\n+\t\tconst char *sha1_file_directory = get_object_directory();\n+\t\tint len = strlen(sha1_file_directory);\n+\t\tbase = xmalloc(len + 60);\n+\t\tsprintf(base, \"%s/pack/pack-1234567890123456789012345678901234567890.idx\", sha1_file_directory);\n+\t\tname = base + len + 11;\n+\t}\n+\n+\tbuf = name;\n+\n+\tfor (i = 0; i < 20; i++) {\n+\t\tunsigned int val = *sha1++;\n+\t\t*buf++ = hex[val >> 4];\n+\t\t*buf++ = hex[val & 0xf];\n+\t}\n+ \n+\treturn base;\n+}\n+\n  struct alternate_object_database *alt_odb;\n\n  /*\n@@ -360,6 +410,14 @@ void unuse_packed_git(struct packed_git\n\n  int use_packed_git(struct packed_git *p)\n  {\n+\tif (!p->pack_size) {\n+\t\tstruct stat st;\n+\t\t// We created the struct before we had the pack\n+\t\tstat(p->pack_name, &st);\n+\t\tif (!S_ISREG(st.st_mode))\n+\t\t\tdie(\"packfile %s not a regular file\", p->pack_name);\n+\t\tp->pack_size = st.st_size;\n+\t}\n  \tif (!p->pack_base) {\n  \t\tint fd;\n  \t\tstruct stat st;\n@@ -387,8 +445,10 @@ int use_packed_git(struct packed_git *p)\n  \t\t * this is cheap.\n  \t\t */\n  \t\tif (memcmp((char*)(p->index_base) + p->index_size - 40,\n-\t\t\t   p->pack_base + p->pack_size - 20, 20))\n+\t\t\t   p->pack_base + p->pack_size - 20, 20)) {\n+\n  \t\t\tdie(\"packfile %s does not match index.\", p->pack_name);\n+\t\t}\n  \t}\n  \tp->pack_last_used = pack_used_ctr++;\n  \tp->pack_use_cnt++;\n@@ -426,6 +486,37 @@ struct packed_git *add_packed_git(char *\n  \treturn p;\n  }\n\n+struct packed_git *parse_pack_index(unsigned char *sha1)\n+{\n+\tstruct packed_git *p;\n+\tunsigned long idx_size;\n+\tvoid *idx_map;\n+\tchar *path = sha1_pack_index_name(sha1);\n+\n+\tif (check_packed_git_idx(path, &idx_size, &idx_map))\n+\t\treturn NULL;\n+\n+\tpath = sha1_pack_name(sha1);\n+\n+\tp = xmalloc(sizeof(*p) + strlen(path) + 2);\n+\tstrcpy(p->pack_name, path);\n+\tp->index_size = idx_size;\n+\tp->pack_size = 0;\n+\tp->index_base = idx_map;\n+\tp->next = NULL;\n+\tp->pack_base = NULL;\n+\tp->pack_last_used = 0;\n+\tp->pack_use_cnt = 0;\n+\tmemcpy(p->sha1, sha1, 20);\n+\treturn p;\n+}\n+\n+void install_packed_git(struct packed_git *pack)\n+{\n+\tpack->next = packed_git;\n+\tpacked_git = pack;\n+}\n+\n  static void prepare_packed_git_one(char *objdir)\n  {\n  \tchar path[PATH_MAX];\n@@ -989,6 +1080,20 @@ static int find_pack_entry(const unsigne\n  \treturn 0;\n  }\n\n+struct packed_git *find_sha1_pack(const unsigned char *sha1, \n+\t\t\t\t  struct packed_git *packs)\n+{\n+\tstruct packed_git *p;\n+\tstruct pack_entry e;\n+\n+\tfor (p = packs; p; p = p->next) {\n+\t\tif (find_pack_entry_one(sha1, &e, p))\n+\t\t\treturn p;\n+\t}\n+\treturn NULL;\n+ \n+}\n+\n  int sha1_object_info(const unsigned char *sha1, char *type, unsigned long *sizep)\n  {\n  \tint status;\n@@ -1335,6 +1440,22 @@ int write_sha1_from_fd(const unsigned ch\n  \treturn 0;\n  }\n\n+int has_pack_index(const unsigned char *sha1)\n+{\n+\tstruct stat st;\n+\tif (stat(sha1_pack_index_name(sha1), &st))\n+\t\treturn 0;\n+\treturn 1;\n+}\n+\n+int has_pack_file(const unsigned char *sha1)\n+{\n+\tstruct stat st;\n+\tif (stat(sha1_pack_name(sha1), &st))\n+\t\treturn 0;\n+\treturn 1;\n+}\n+\n  int has_sha1_pack(const unsigned char *sha1)\n  {\n  \tstruct pack_entry e;\n"},{"id":"6674","messageId":"Pine.LNX.4.62.0507311610270.23721@iabervon.org","threadId":"1380","inReplyTo":"Pine.LNX.4.62.0507311600040.23721@iabervon.org","subject":"[PATCH 2/2] Support downloading packs by HTTP","fromName":"","fromEmail":"barkalow@iabervon.org","sentAt":"2005-07-31T20:10:53Z","receivedAt":"2005-07-31T20:10:53Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"This adds support to http-pull for finding the list of pack files\navailable on the server, downloading the index files for those pack\nfiles, and downloading pack files when they contain needed objects not\navailable individually. It retains the index files even if the pack\nfiles were not needed, but downloads the list of pack files once per\nrun if an object is not found separately.\n\nSigned-off-by: Daniel Barkalow <barkalow@iabervon.org>\n---\n\n  http-pull.c |  181 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n  1 files changed, 175 insertions(+), 6 deletions(-)\n\ndff0b76c4a2efbb8407778a1da6dc2ea2ca1458f\ndiff --git a/http-pull.c b/http-pull.c\n--- a/http-pull.c\n+++ b/http-pull.c\n@@ -33,7 +33,8 @@ struct buffer\n  };\n\n  static size_t fwrite_buffer(void *ptr, size_t eltsize, size_t nmemb,\n-                            struct buffer *buffer) {\n+                            struct buffer *buffer)\n+{\n          size_t size = eltsize * nmemb;\n          if (size > buffer->size - buffer->posn)\n                  size = buffer->size - buffer->posn;\n@@ -42,8 +43,9 @@ static size_t fwrite_buffer(void *ptr, s\n          return size;\n  }\n\n-static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb, \n-\t\t\t       void *data) {\n+static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,\n+\t\t\t       void *data)\n+{\n  \tunsigned char expn[4096];\n  \tsize_t size = eltsize * nmemb;\n  \tint posn = 0;\n@@ -65,6 +67,168 @@ static size_t fwrite_sha1_file(void *ptr\n  \treturn size;\n  }\n\n+static int got_indices = 0;\n+\n+static struct packed_git *packs = NULL;\n+\n+static int fetch_index(unsigned char *sha1)\n+{\n+\tchar *filename;\n+\tchar *url;\n+\n+\tFILE *indexfile;\n+\n+\tif (has_pack_index(sha1))\n+\t\treturn 0;\n+\n+\tif (get_verbosely)\n+\t\tfprintf(stderr, \"Getting index for pack %s\\n\",\n+\t\t\tsha1_to_hex(sha1));\n+ \n+\turl = xmalloc(strlen(base) + 64);\n+\tsprintf(url, \"%s/objects/pack/pack-%s.idx\",\n+\t\tbase, sha1_to_hex(sha1));\n+ \n+\tfilename = sha1_pack_index_name(sha1);\n+\tindexfile = fopen(filename, \"w\");\n+\tif (!indexfile)\n+\t\treturn error(\"Unable to open local file %s for pack index\",\n+\t\t\t     filename);\n+\n+\tcurl_easy_setopt(curl, CURLOPT_FILE, indexfile);\n+\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n+\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+ \n+\tif (curl_easy_perform(curl)) {\n+\t\tfclose(indexfile);\n+\t\treturn error(\"Unable to get pack index %s\", url);\n+\t}\n+\n+\tfclose(indexfile);\n+\treturn 0;\n+}\n+\n+static int setup_index(unsigned char *sha1)\n+{\n+\tstruct packed_git *new_pack;\n+\tif (has_pack_file(sha1))\n+\t\treturn 0; // don't list this as something we can get\n+\n+\tif (fetch_index(sha1))\n+\t\treturn -1;\n+\n+\tnew_pack = parse_pack_index(sha1);\n+\tnew_pack->next = packs;\n+\tpacks = new_pack;\n+\treturn 0;\n+}\n+\n+static int fetch_indices(void)\n+{\n+\tunsigned char sha1[20];\n+\tchar *url;\n+\tstruct buffer buffer;\n+\tchar *data;\n+\tint i = 0;\n+\n+\tif (got_indices)\n+\t\treturn 0;\n+\n+\tdata = xmalloc(4096);\n+\tbuffer.size = 4096;\n+\tbuffer.posn = 0;\n+\tbuffer.buffer = data;\n+\n+\tif (get_verbosely)\n+\t\tfprintf(stderr, \"Getting pack list\\n\");\n+ \n+\turl = xmalloc(strlen(base) + 21);\n+\tsprintf(url, \"%s/objects/info/packs\", base);\n+\n+\tcurl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n+\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+ \n+\tif (curl_easy_perform(curl)) {\n+\t\treturn error(\"Unable to get pack index %s\", url);\n+\t}\n+\n+\tdo {\n+\t\tswitch (data[i]) {\n+\t\tcase 'P':\n+\t\t\ti++;\n+\t\t\tif (i + 52 < buffer.posn &&\n+\t\t\t    !strncmp(data + i, \" pack-\", 6) &&\n+\t\t\t    !strncmp(data + i + 46, \".pack\\n\", 6)) {\n+\t\t\t\tget_sha1_hex(data + i + 6, sha1);\n+\t\t\t\tsetup_index(sha1);\n+\t\t\t\ti += 51;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\tdefault:\n+\t\t\twhile (data[i] != '\\n')\n+\t\t\t\ti++;\n+\t\t}\n+\t\ti++;\n+\t} while (i < buffer.posn);\n+\n+\tgot_indices = 1;\n+\treturn 0;\n+}\n+\n+static int fetch_pack(unsigned char *sha1)\n+{\n+\tchar *url;\n+\tstruct packed_git *target;\n+\tstruct packed_git **lst;\n+\tFILE *packfile;\n+\tchar *filename;\n+\n+\tif (fetch_indices())\n+\t\treturn -1;\n+\ttarget = find_sha1_pack(sha1, packs);\n+\tif (!target)\n+\t\treturn error(\"Couldn't get %s: not separate or in any pack\",\n+\t\t\t     sha1_to_hex(sha1));\n+\n+\tif (get_verbosely) {\n+\t\tfprintf(stderr, \"Getting pack %s\\n\",\n+\t\t\tsha1_to_hex(target->sha1));\n+\t\tfprintf(stderr, \" which contains %s\\n\",\n+\t\t\tsha1_to_hex(sha1));\n+\t}\n+\n+\turl = xmalloc(strlen(base) + 65);\n+\tsprintf(url, \"%s/objects/pack/pack-%s.pack\",\n+\t\tbase, sha1_to_hex(target->sha1));\n+\n+\tfilename = sha1_pack_name(target->sha1);\n+\tpackfile = fopen(filename, \"w\");\n+\tif (!packfile)\n+\t\treturn error(\"Unable to open local file %s for pack\",\n+\t\t\t     filename);\n+\n+\tcurl_easy_setopt(curl, CURLOPT_FILE, packfile);\n+\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n+\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+ \n+\tif (curl_easy_perform(curl)) {\n+\t\tfclose(packfile);\n+\t\treturn error(\"Unable to get pack file %s\", url);\n+\t}\n+\n+\tfclose(packfile);\n+\n+\tlst = &packs;\n+\twhile (*lst != target)\n+\t\tlst = &((*lst)->next);\n+\t*lst = (*lst)->next;\n+\n+\tinstall_packed_git(target);\n+\n+\treturn 0;\n+}\n+\n  int fetch(unsigned char *sha1)\n  {\n  \tchar *hex = sha1_to_hex(sha1);\n@@ -76,7 +240,7 @@ int fetch(unsigned char *sha1)\n  \tlocal = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);\n\n  \tif (local < 0)\n-\t\treturn error(\"Couldn't open %s\\n\", filename);\n+\t\treturn error(\"Couldn't open local object %s\\n\", filename);\n\n  \tmemset(&stream, 0, sizeof(stream));\n\n@@ -84,6 +248,7 @@ int fetch(unsigned char *sha1)\n\n  \tSHA1_Init(&c);\n\n+\tcurl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);\n  \tcurl_easy_setopt(curl, CURLOPT_FILE, NULL);\n  \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n\n@@ -99,8 +264,12 @@ int fetch(unsigned char *sha1)\n\n  \tcurl_easy_setopt(curl, CURLOPT_URL, url);\n\n-\tif (curl_easy_perform(curl))\n-\t\treturn error(\"Couldn't get %s for %s\\n\", url, hex);\n+\tif (curl_easy_perform(curl)) {\n+\t\tunlink(filename);\n+\t\tif (fetch_pack(sha1))\n+\t\t\treturn error(\"Tried %s\", url);\n+\t\treturn 0;\n+\t}\n\n  \tclose(local);\n  \tinflateEnd(&stream);\n"},{"id":"6685","messageId":"7vvf2qbkvr.fsf@assigned-by-dhcp.cox.net","threadId":"1380","inReplyTo":"Pine.LNX.4.62.0507311609420.23721@iabervon.org","subject":"Re: [PATCH 1/2] Functions for managing the set of packs the library is using","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-07-31T23:45:12Z","receivedAt":"2005-07-31T23:45:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel, I would really have liked to merge this immediately, but\nsomehow the patch is whitespace damaged.  Depressingly enough,\nalmost all patches I got from various people today had different\nwhitespace damages, and I started to suspect if there is\nsomething wrong on my end, but it does not appear to be the\ncase.  The copies found in the usual archive I check show the\nsame problems as the copies I got directly in my mailbox have.\n\nCould you resend these two if it is not too much trouble for\nyou?  \n\nThanks for fixing the curl_easy_setopt() screwup.  I should have\nbeen more careful.  I've already hand-merged it and will be\npushing it out shortly, so there is no need to resend that one.\n"},{"id":"6691","messageId":"Pine.LNX.4.62.0507312053010.23721@iabervon.org","threadId":"1380","inReplyTo":"Pine.LNX.4.62.0507311600040.23721@iabervon.org","subject":"[PATCH 1/2] Functions for managing the set of packs the library is using (whitespace fixed)","fromName":"","fromEmail":"barkalow@iabervon.org","sentAt":"2005-08-01T00:53:44Z","receivedAt":"2005-08-01T00:53:44Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"This adds support for reading an uninstalled index, and installing a\npack file that was added while the program was running, as well as\nfunctions for determining where to put the file.\n\nSigned-off-by: Daniel Barkalow <barkalow@iabervon.org>\n---\n\n cache.h     |   13 ++++++\n sha1_file.c |  123 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 135 insertions(+), 1 deletions(-)\n\n20fcc8f66a6780cf9bbd2fc2ba3b918c33696a67\ndiff --git a/cache.h b/cache.h\n--- a/cache.h\n+++ b/cache.h\n@@ -172,6 +172,8 @@ extern void rollback_index_file(struct c\n extern char *mkpath(const char *fmt, ...);\n extern char *git_path(const char *fmt, ...);\n extern char *sha1_file_name(const unsigned char *sha1);\n+extern char *sha1_pack_name(const unsigned char *sha1);\n+extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n int safe_create_leading_directories(char *path);\n \n@@ -200,6 +202,9 @@ extern int write_sha1_to_fd(int fd, cons\n extern int has_sha1_pack(const unsigned char *sha1);\n extern int has_sha1_file(const unsigned char *sha1);\n \n+extern int has_pack_file(const unsigned char *sha1);\n+extern int has_pack_index(const unsigned char *sha1);\n+\n /* Convert to/from hex/sha1 representation */\n extern int get_sha1(const char *str, unsigned char *sha1);\n extern int get_sha1_hex(const char *hex, unsigned char *sha1);\n@@ -276,6 +281,7 @@ extern struct packed_git {\n \tvoid *pack_base;\n \tunsigned int pack_last_used;\n \tunsigned int pack_use_cnt;\n+\tunsigned char sha1[20];\n \tchar pack_name[0]; /* something like \".git/objects/pack/xxxxx.pack\" */\n } *packed_git;\n \n@@ -298,7 +304,14 @@ extern int path_match(const char *path, \n extern int get_ack(int fd, unsigned char *result_sha1);\n extern struct ref **get_remote_heads(int in, struct ref **list, int nr_match, char **match);\n \n+extern struct packed_git *parse_pack_index(unsigned char *sha1);\n+\n extern void prepare_packed_git(void);\n+extern void install_packed_git(struct packed_git *pack);\n+\n+extern struct packed_git *find_sha1_pack(const unsigned char *sha1, \n+\t\t\t\t\t struct packed_git *packs);\n+\n extern int use_packed_git(struct packed_git *);\n extern void unuse_packed_git(struct packed_git *);\n extern struct packed_git *add_packed_git(char *, int);\ndiff --git a/sha1_file.c b/sha1_file.c\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -200,6 +200,56 @@ char *sha1_file_name(const unsigned char\n \treturn base;\n }\n \n+char *sha1_pack_name(const unsigned char *sha1)\n+{\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\tstatic char *name, *base, *buf;\n+\tint i;\n+\n+\tif (!base) {\n+\t\tconst char *sha1_file_directory = get_object_directory();\n+\t\tint len = strlen(sha1_file_directory);\n+\t\tbase = xmalloc(len + 60);\n+\t\tsprintf(base, \"%s/pack/pack-1234567890123456789012345678901234567890.pack\", sha1_file_directory);\n+\t\tname = base + len + 11;\n+\t}\n+\n+\tbuf = name;\n+\n+\tfor (i = 0; i < 20; i++) {\n+\t\tunsigned int val = *sha1++;\n+\t\t*buf++ = hex[val >> 4];\n+\t\t*buf++ = hex[val & 0xf];\n+\t}\n+\t\n+\treturn base;\n+}\n+\n+char *sha1_pack_index_name(const unsigned char *sha1)\n+{\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\tstatic char *name, *base, *buf;\n+\tint i;\n+\n+\tif (!base) {\n+\t\tconst char *sha1_file_directory = get_object_directory();\n+\t\tint len = strlen(sha1_file_directory);\n+\t\tbase = xmalloc(len + 60);\n+\t\tsprintf(base, \"%s/pack/pack-1234567890123456789012345678901234567890.idx\", sha1_file_directory);\n+\t\tname = base + len + 11;\n+\t}\n+\n+\tbuf = name;\n+\n+\tfor (i = 0; i < 20; i++) {\n+\t\tunsigned int val = *sha1++;\n+\t\t*buf++ = hex[val >> 4];\n+\t\t*buf++ = hex[val & 0xf];\n+\t}\n+\t\n+\treturn base;\n+}\n+\n struct alternate_object_database *alt_odb;\n \n /*\n@@ -360,6 +410,14 @@ void unuse_packed_git(struct packed_git \n \n int use_packed_git(struct packed_git *p)\n {\n+\tif (!p->pack_size) {\n+\t\tstruct stat st;\n+\t\t// We created the struct before we had the pack\n+\t\tstat(p->pack_name, &st);\n+\t\tif (!S_ISREG(st.st_mode))\n+\t\t\tdie(\"packfile %s not a regular file\", p->pack_name);\n+\t\tp->pack_size = st.st_size;\n+\t}\n \tif (!p->pack_base) {\n \t\tint fd;\n \t\tstruct stat st;\n@@ -387,8 +445,10 @@ int use_packed_git(struct packed_git *p)\n \t\t * this is cheap.\n \t\t */\n \t\tif (memcmp((char*)(p->index_base) + p->index_size - 40,\n-\t\t\t   p->pack_base + p->pack_size - 20, 20))\n+\t\t\t   p->pack_base + p->pack_size - 20, 20)) {\n+\t\t\t      \n \t\t\tdie(\"packfile %s does not match index.\", p->pack_name);\n+\t\t}\n \t}\n \tp->pack_last_used = pack_used_ctr++;\n \tp->pack_use_cnt++;\n@@ -426,6 +486,37 @@ struct packed_git *add_packed_git(char *\n \treturn p;\n }\n \n+struct packed_git *parse_pack_index(unsigned char *sha1)\n+{\n+\tstruct packed_git *p;\n+\tunsigned long idx_size;\n+\tvoid *idx_map;\n+\tchar *path = sha1_pack_index_name(sha1);\n+\n+\tif (check_packed_git_idx(path, &idx_size, &idx_map))\n+\t\treturn NULL;\n+\n+\tpath = sha1_pack_name(sha1);\n+\n+\tp = xmalloc(sizeof(*p) + strlen(path) + 2);\n+\tstrcpy(p->pack_name, path);\n+\tp->index_size = idx_size;\n+\tp->pack_size = 0;\n+\tp->index_base = idx_map;\n+\tp->next = NULL;\n+\tp->pack_base = NULL;\n+\tp->pack_last_used = 0;\n+\tp->pack_use_cnt = 0;\n+\tmemcpy(p->sha1, sha1, 20);\n+\treturn p;\n+}\n+\n+void install_packed_git(struct packed_git *pack)\n+{\n+\tpack->next = packed_git;\n+\tpacked_git = pack;\n+}\n+\n static void prepare_packed_git_one(char *objdir)\n {\n \tchar path[PATH_MAX];\n@@ -989,6 +1080,20 @@ static int find_pack_entry(const unsigne\n \treturn 0;\n }\n \n+struct packed_git *find_sha1_pack(const unsigned char *sha1, \n+\t\t\t\t  struct packed_git *packs)\n+{\n+\tstruct packed_git *p;\n+\tstruct pack_entry e;\n+\n+\tfor (p = packs; p; p = p->next) {\n+\t\tif (find_pack_entry_one(sha1, &e, p))\n+\t\t\treturn p;\n+\t}\n+\treturn NULL;\n+\t\n+}\n+\n int sha1_object_info(const unsigned char *sha1, char *type, unsigned long *sizep)\n {\n \tint status;\n@@ -1335,6 +1440,22 @@ int write_sha1_from_fd(const unsigned ch\n \treturn 0;\n }\n \n+int has_pack_index(const unsigned char *sha1)\n+{\n+\tstruct stat st;\n+\tif (stat(sha1_pack_index_name(sha1), &st))\n+\t\treturn 0;\n+\treturn 1;\n+}\n+\n+int has_pack_file(const unsigned char *sha1)\n+{\n+\tstruct stat st;\n+\tif (stat(sha1_pack_name(sha1), &st))\n+\t\treturn 0;\n+\treturn 1;\n+}\n+\n int has_sha1_pack(const unsigned char *sha1)\n {\n \tstruct pack_entry e;\n"},{"id":"6692","messageId":"Pine.LNX.4.62.0507312053470.23721@iabervon.org","threadId":"1380","inReplyTo":"Pine.LNX.4.62.0507311600040.23721@iabervon.org","subject":"[PATCH 2/2] Support downloading packs by HTTP (whitespace fixed)","fromName":"","fromEmail":"barkalow@iabervon.org","sentAt":"2005-08-01T00:54:17Z","receivedAt":"2005-08-01T00:54:17Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"This adds support to http-pull for finding the list of pack files\navailable on the server, downloading the index files for those pack\nfiles, and downloading pack files when they contain needed objects not\navailable individually. It retains the index files even if the pack\nfiles were not needed, but downloads the list of pack files once per\nrun if an object is not found separately.\n\nSigned-off-by: Daniel Barkalow <barkalow@iabervon.org>\n---\n\n http-pull.c |  181 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 175 insertions(+), 6 deletions(-)\n\ndff0b76c4a2efbb8407778a1da6dc2ea2ca1458f\ndiff --git a/http-pull.c b/http-pull.c\n--- a/http-pull.c\n+++ b/http-pull.c\n@@ -33,7 +33,8 @@ struct buffer\n };\n \n static size_t fwrite_buffer(void *ptr, size_t eltsize, size_t nmemb,\n-                            struct buffer *buffer) {\n+                            struct buffer *buffer)\n+{\n         size_t size = eltsize * nmemb;\n         if (size > buffer->size - buffer->posn)\n                 size = buffer->size - buffer->posn;\n@@ -42,8 +43,9 @@ static size_t fwrite_buffer(void *ptr, s\n         return size;\n }\n \n-static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb, \n-\t\t\t       void *data) {\n+static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,\n+\t\t\t       void *data)\n+{\n \tunsigned char expn[4096];\n \tsize_t size = eltsize * nmemb;\n \tint posn = 0;\n@@ -65,6 +67,168 @@ static size_t fwrite_sha1_file(void *ptr\n \treturn size;\n }\n \n+static int got_indices = 0;\n+\n+static struct packed_git *packs = NULL;\n+\n+static int fetch_index(unsigned char *sha1)\n+{\n+\tchar *filename;\n+\tchar *url;\n+\n+\tFILE *indexfile;\n+\n+\tif (has_pack_index(sha1))\n+\t\treturn 0;\n+\n+\tif (get_verbosely)\n+\t\tfprintf(stderr, \"Getting index for pack %s\\n\",\n+\t\t\tsha1_to_hex(sha1));\n+\t\n+\turl = xmalloc(strlen(base) + 64);\n+\tsprintf(url, \"%s/objects/pack/pack-%s.idx\",\n+\t\tbase, sha1_to_hex(sha1));\n+\t\n+\tfilename = sha1_pack_index_name(sha1);\n+\tindexfile = fopen(filename, \"w\");\n+\tif (!indexfile)\n+\t\treturn error(\"Unable to open local file %s for pack index\",\n+\t\t\t     filename);\n+\n+\tcurl_easy_setopt(curl, CURLOPT_FILE, indexfile);\n+\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n+\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\t\n+\tif (curl_easy_perform(curl)) {\n+\t\tfclose(indexfile);\n+\t\treturn error(\"Unable to get pack index %s\", url);\n+\t}\n+\n+\tfclose(indexfile);\n+\treturn 0;\n+}\n+\n+static int setup_index(unsigned char *sha1)\n+{\n+\tstruct packed_git *new_pack;\n+\tif (has_pack_file(sha1))\n+\t\treturn 0; // don't list this as something we can get\n+\n+\tif (fetch_index(sha1))\n+\t\treturn -1;\n+\n+\tnew_pack = parse_pack_index(sha1);\n+\tnew_pack->next = packs;\n+\tpacks = new_pack;\n+\treturn 0;\n+}\n+\n+static int fetch_indices(void)\n+{\n+\tunsigned char sha1[20];\n+\tchar *url;\n+\tstruct buffer buffer;\n+\tchar *data;\n+\tint i = 0;\n+\n+\tif (got_indices)\n+\t\treturn 0;\n+\n+\tdata = xmalloc(4096);\n+\tbuffer.size = 4096;\n+\tbuffer.posn = 0;\n+\tbuffer.buffer = data;\n+\n+\tif (get_verbosely)\n+\t\tfprintf(stderr, \"Getting pack list\\n\");\n+\t\n+\turl = xmalloc(strlen(base) + 21);\n+\tsprintf(url, \"%s/objects/info/packs\", base);\n+\n+\tcurl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n+\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\t\n+\tif (curl_easy_perform(curl)) {\n+\t\treturn error(\"Unable to get pack index %s\", url);\n+\t}\n+\n+\tdo {\n+\t\tswitch (data[i]) {\n+\t\tcase 'P':\n+\t\t\ti++;\n+\t\t\tif (i + 52 < buffer.posn &&\n+\t\t\t    !strncmp(data + i, \" pack-\", 6) &&\n+\t\t\t    !strncmp(data + i + 46, \".pack\\n\", 6)) {\n+\t\t\t\tget_sha1_hex(data + i + 6, sha1);\n+\t\t\t\tsetup_index(sha1);\n+\t\t\t\ti += 51;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\tdefault:\n+\t\t\twhile (data[i] != '\\n')\n+\t\t\t\ti++;\n+\t\t}\n+\t\ti++;\n+\t} while (i < buffer.posn);\n+\n+\tgot_indices = 1;\n+\treturn 0;\n+}\n+\n+static int fetch_pack(unsigned char *sha1)\n+{\n+\tchar *url;\n+\tstruct packed_git *target;\n+\tstruct packed_git **lst;\n+\tFILE *packfile;\n+\tchar *filename;\n+\n+\tif (fetch_indices())\n+\t\treturn -1;\n+\ttarget = find_sha1_pack(sha1, packs);\n+\tif (!target)\n+\t\treturn error(\"Couldn't get %s: not separate or in any pack\",\n+\t\t\t     sha1_to_hex(sha1));\n+\n+\tif (get_verbosely) {\n+\t\tfprintf(stderr, \"Getting pack %s\\n\",\n+\t\t\tsha1_to_hex(target->sha1));\n+\t\tfprintf(stderr, \" which contains %s\\n\",\n+\t\t\tsha1_to_hex(sha1));\n+\t}\n+\n+\turl = xmalloc(strlen(base) + 65);\n+\tsprintf(url, \"%s/objects/pack/pack-%s.pack\",\n+\t\tbase, sha1_to_hex(target->sha1));\n+\n+\tfilename = sha1_pack_name(target->sha1);\n+\tpackfile = fopen(filename, \"w\");\n+\tif (!packfile)\n+\t\treturn error(\"Unable to open local file %s for pack\",\n+\t\t\t     filename);\n+\n+\tcurl_easy_setopt(curl, CURLOPT_FILE, packfile);\n+\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n+\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\t\n+\tif (curl_easy_perform(curl)) {\n+\t\tfclose(packfile);\n+\t\treturn error(\"Unable to get pack file %s\", url);\n+\t}\n+\n+\tfclose(packfile);\n+\n+\tlst = &packs;\n+\twhile (*lst != target)\n+\t\tlst = &((*lst)->next);\n+\t*lst = (*lst)->next;\n+\n+\tinstall_packed_git(target);\n+\n+\treturn 0;\n+}\n+\n int fetch(unsigned char *sha1)\n {\n \tchar *hex = sha1_to_hex(sha1);\n@@ -76,7 +240,7 @@ int fetch(unsigned char *sha1)\n \tlocal = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);\n \n \tif (local < 0)\n-\t\treturn error(\"Couldn't open %s\\n\", filename);\n+\t\treturn error(\"Couldn't open local object %s\\n\", filename);\n \n \tmemset(&stream, 0, sizeof(stream));\n \n@@ -84,6 +248,7 @@ int fetch(unsigned char *sha1)\n \n \tSHA1_Init(&c);\n \n+\tcurl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);\n \tcurl_easy_setopt(curl, CURLOPT_FILE, NULL);\n \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n \n@@ -99,8 +264,12 @@ int fetch(unsigned char *sha1)\n \n \tcurl_easy_setopt(curl, CURLOPT_URL, url);\n \n-\tif (curl_easy_perform(curl))\n-\t\treturn error(\"Couldn't get %s for %s\\n\", url, hex);\n+\tif (curl_easy_perform(curl)) {\n+\t\tunlink(filename);\n+\t\tif (fetch_pack(sha1))\n+\t\t\treturn error(\"Tried %s\", url);\n+\t\treturn 0;\n+\t}\n \n \tclose(local);\n \tinflateEnd(&stream);\n"},{"id":"6702","messageId":"7vpssyxeag.fsf@assigned-by-dhcp.cox.net","threadId":"1380","inReplyTo":"Pine.LNX.4.62.0507312053010.23721@iabervon.org","subject":"Re: [PATCH 1/2] Functions for managing the set of packs the library is using (whitespace fixed)","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-08-01T08:16:39Z","receivedAt":"2005-08-01T08:16:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"barkalow@iabervon.org writes:\n\n> This adds support for reading an uninstalled index, and installing a\n> pack file that was added while the program was running, as well as\n> functions for determining where to put the file.\n\nThanks.  Applied and pushed out.\n\nI removed my own version of stop-gap hack; git-fetch-script\nnow directly calls git-http-pull.\n"}]}