{"thread":{"id":"1966","subject":"[PATCH] HTTP partial transfer support for object, pack, and index transfers","startedAt":"2005-09-28T17:14:04Z","lastAt":"2005-10-01T07:17:21Z","messageCount":4,"participants":["Nick Hengeveld","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"9433","messageId":"20050928171404.GA15593@reactrix.com","threadId":"1966","inReplyTo":null,"subject":"[PATCH] HTTP partial transfer support for object, pack, and index transfers","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-09-28T17:14:04Z","receivedAt":"2005-09-28T17:14:04Z","isPatch":true,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"HTTP partial transfer support for object, pack, and index transfers\n\nSigned-off-by: Nick Hengeveld <nickh@reactrix.com>\n\n\n---\n\n http-fetch.c |  185 ++++++++++++++++++++++++++++++++++++++++++++++++++--------\n 1 files changed, 161 insertions(+), 24 deletions(-)\n\n492f6dfd5f0e7030fe96b3646a07a9adb1a7a2d0\ndiff --git a/http-fetch.c b/http-fetch.c\n--- a/http-fetch.c\n+++ b/http-fetch.c\n@@ -13,8 +13,12 @@\n #define curl_global_init(a) do { /* nothing */ } while(0)\n #endif\n \n+#define PREV_BUF_SIZE 4096\n+#define RANGE_HEADER_SIZE 30\n+\n static CURL *curl;\n static struct curl_slist *no_pragma_header;\n+static struct curl_slist *no_range_header;\n \n static char *initial_base;\n \n@@ -82,12 +86,37 @@ void prefetch(unsigned char *sha1)\n {\n }\n \n+int relink_or_rename(char *old, char *new) {\n+\tint ret;\n+\n+\tret = link(old, new);\n+\tif (ret < 0) {\n+\t\t/* Same Coda hack as in write_sha1_file(sha1_file.c) */\n+\t\tret = errno;\n+\t\tif (ret == EXDEV && !rename(old, new))\n+\t\t\treturn 0;\n+\t}\n+\tunlink(old);\n+\tif (ret) {\n+\t\tif (ret != EEXIST)\n+\t\t\treturn ret;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static int got_alternates = 0;\n \n static int fetch_index(struct alt_base *repo, unsigned char *sha1)\n {\n \tchar *filename;\n \tchar *url;\n+\tchar tmpfile[PATH_MAX];\n+\tint ret;\n+\tlong prev_posn = 0;\n+\tchar range[RANGE_HEADER_SIZE];\n+\tstruct curl_slist *range_header = NULL;\n+\tCURLcode curl_result;\n \n \tFILE *indexfile;\n \n@@ -103,7 +132,8 @@ static int fetch_index(struct alt_base *\n \t\trepo->base, sha1_to_hex(sha1));\n \t\n \tfilename = sha1_pack_index_name(sha1);\n-\tindexfile = fopen(filename, \"w\");\n+\tsnprintf(tmpfile, sizeof(tmpfile), \"%s.temp\", filename);\n+\tindexfile = fopen(tmpfile, \"a\");\n \tif (!indexfile)\n \t\treturn error(\"Unable to open local file %s for pack index\",\n \t\t\t     filename);\n@@ -113,12 +143,35 @@ static int fetch_index(struct alt_base *\n \tcurl_easy_setopt(curl, CURLOPT_URL, url);\n \tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);\n \t\n-\tif (curl_easy_perform(curl)) {\n+\t/* If there is data present from a previous transfer attempt,\n+\t   resume where it left off */\n+\tprev_posn = ftell(indexfile);\n+\tif (prev_posn>0) {\n+\t\tif (get_verbosely)\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"Resuming fetch of index for pack %s at byte %ld\\n\",\n+\t\t\t\tsha1_to_hex(sha1), prev_posn);\n+\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n+\t\trange_header = curl_slist_append(range_header, range);\n+\t\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, range_header);\n+\t}\n+\n+\t/* Clear out the Range: header after performing the request, so\n+\t   other curl requests don't inherit inappropriate header data */\n+\tcurl_result = curl_easy_perform(curl);\n+\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);\n+\tif (curl_result != 0) {\n \t\tfclose(indexfile);\n \t\treturn error(\"Unable to get pack index %s\", url);\n \t}\n \n \tfclose(indexfile);\n+\n+\tret = relink_or_rename(tmpfile, filename);\n+\tif (ret)\n+\t\treturn error(\"unable to write index filename %s: %s\",\n+\t\t\t     filename, strerror(ret));\n+\n \treturn 0;\n }\n \n@@ -299,6 +352,12 @@ static int fetch_pack(struct alt_base *r\n \tstruct packed_git **lst;\n \tFILE *packfile;\n \tchar *filename;\n+\tchar tmpfile[PATH_MAX];\n+\tint ret;\n+\tlong prev_posn = 0;\n+\tchar range[RANGE_HEADER_SIZE];\n+\tstruct curl_slist *range_header = NULL;\n+\tCURLcode curl_result;\n \n \tif (fetch_indices(repo))\n \t\treturn -1;\n@@ -318,7 +377,8 @@ static int fetch_pack(struct alt_base *r\n \t\trepo->base, sha1_to_hex(target->sha1));\n \n \tfilename = sha1_pack_name(target->sha1);\n-\tpackfile = fopen(filename, \"w\");\n+\tsnprintf(tmpfile, sizeof(tmpfile), \"%s.temp\", filename);\n+\tpackfile = fopen(tmpfile, \"a\");\n \tif (!packfile)\n \t\treturn error(\"Unable to open local file %s for pack\",\n \t\t\t     filename);\n@@ -327,14 +387,36 @@ static int fetch_pack(struct alt_base *r\n \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n \tcurl_easy_setopt(curl, CURLOPT_URL, url);\n \tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);\n-\t\n-\tif (curl_easy_perform(curl)) {\n+\n+\t/* If there is data present from a previous transfer attempt,\n+\t   resume where it left off */\n+\tprev_posn = ftell(packfile);\n+\tif (prev_posn>0) {\n+\t\tif (get_verbosely)\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"Resuming fetch of pack %s at byte %ld\\n\",\n+\t\t\t\tsha1_to_hex(target->sha1), prev_posn);\n+\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n+\t\trange_header = curl_slist_append(range_header, range);\n+\t\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, range_header);\n+\t}\n+\n+\t/* Clear out the Range: header after performing the request, so\n+\t   other curl requests don't inherit inappropriate header data */\n+\tcurl_result = curl_easy_perform(curl);\n+\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);\n+\tif (curl_result != 0) {\n \t\tfclose(packfile);\n \t\treturn error(\"Unable to get pack file %s\", url);\n \t}\n \n \tfclose(packfile);\n \n+\tret = relink_or_rename(tmpfile, filename);\n+\tif (ret)\n+\t\treturn error(\"unable to write pack filename %s: %s\",\n+\t\t\t     filename, strerror(ret));\n+\n \tlst = &repo->packs;\n \twhile (*lst != target)\n \t\tlst = &((*lst)->next);\n@@ -351,14 +433,29 @@ int fetch_object(struct alt_base *repo, \n \tchar *filename = sha1_file_name(sha1);\n \tunsigned char real_sha1[20];\n \tchar tmpfile[PATH_MAX];\n+\tchar prevfile[PATH_MAX];\n \tint ret;\n \tchar *url;\n \tchar *posn;\n+\tint prevlocal;\n+\tunsigned char prev_buf[PREV_BUF_SIZE];\n+\tssize_t prev_read = 0;\n+\tlong prev_posn = 0;\n+\tchar range[RANGE_HEADER_SIZE];\n+\tstruct curl_slist *range_header = NULL;\n+\tCURLcode curl_result;\n+\n+\tsnprintf(tmpfile, sizeof(tmpfile), \"%s.temp\", filename);\n+\tsnprintf(prevfile, sizeof(prevfile), \"%s.prev\", filename);\n+\tunlink(prevfile);\n+\trename(tmpfile, prevfile);\n+\tunlink(tmpfile);\n+\n+\tlocal = open(tmpfile, O_WRONLY | O_CREAT | O_EXCL, 0666);\n \n-\tsnprintf(tmpfile, sizeof(tmpfile), \"%s/obj_XXXXXX\",\n-\t\t get_object_directory());\n+\t/* Note: if another instance starts now, it will turn our new\n+\t   tmpfile into its prevfile. */\n \n-\tlocal = mkstemp(tmpfile);\n \tif (local < 0)\n \t\treturn error(\"Couldn't create temporary file %s for %s: %s\\n\",\n \t\t\t     tmpfile, filename, strerror(errno));\n@@ -386,8 +483,56 @@ int fetch_object(struct alt_base *repo, \n \n \tcurl_easy_setopt(curl, CURLOPT_URL, url);\n \n-\tif (curl_easy_perform(curl)) {\n-\t\tunlink(filename);\n+\t/* If a previous temp file is present, process what was already\n+\t   fetched. */\n+\tprevlocal = open(prevfile, O_RDONLY);\n+\tif (prevlocal != -1) {\n+\t\tdo {\n+\t\t\tprev_read = read(prevlocal, prev_buf, PREV_BUF_SIZE);\n+\t\t\tif (prev_read>0) {\n+\t\t\t\tif (fwrite_sha1_file(prev_buf,\n+\t\t\t\t\t\t     1,\n+\t\t\t\t\t\t     prev_read,\n+\t\t\t\t\t\t     NULL) == prev_read) {\n+\t\t\t\t\tprev_posn += prev_read;\n+\t\t\t\t} else {\n+\t\t\t\t\tprev_read = -1;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t} while (prev_read > 0);\n+\t\tclose(prevlocal);\n+\t}\n+\tunlink(prevfile);\n+\n+\t/* Reset inflate/SHA1 if there was an error reading the previous temp\n+\t   file; also rewind to the beginning of the local file. */\n+\tif (prev_read == -1) {\n+\t\tmemset(&stream, 0, sizeof(stream));\n+\t\tinflateInit(&stream);\n+\t\tSHA1_Init(&c);\n+\t\tif (prev_posn>0) {\n+\t\t\tprev_posn = 0;\n+\t\t\tlseek(local, SEEK_SET, 0);\n+\t\t}\n+\t}\n+\n+\t/* If we have successfully processed data from a previous fetch\n+\t   attempt, only fetch the data we don't already have. */\n+\tif (prev_posn>0) {\n+\t\tif (get_verbosely)\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"Resuming fetch of object %s at byte %ld\\n\",\n+\t\t\t\thex, prev_posn);\n+\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n+\t\trange_header = curl_slist_append(range_header, range);\n+\t\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, range_header);\n+\t}\n+\n+\t/* Clear out the Range: header after performing the request, so\n+\t   other curl requests don't inherit inappropriate header data */\n+\tcurl_result = curl_easy_perform(curl);\n+\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);\n+\tif (curl_result != 0) {\n \t\treturn -1;\n \t}\n \n@@ -403,20 +548,11 @@ int fetch_object(struct alt_base *repo, \n \t\tunlink(tmpfile);\n \t\treturn error(\"File %s has bad hash\\n\", hex);\n \t}\n-\tret = link(tmpfile, filename);\n-\tif (ret < 0) {\n-\t\t/* Same Coda hack as in write_sha1_file(sha1_file.c) */\n-\t\tret = errno;\n-\t\tif (ret == EXDEV && !rename(tmpfile, filename))\n-\t\t\tgoto out;\n-\t}\n-\tunlink(tmpfile);\n-\tif (ret) {\n-\t\tif (ret != EEXIST)\n-\t\t\treturn error(\"unable to write sha1 filename %s: %s\",\n-\t\t\t\t     filename, strerror(ret));\n-\t}\n- out:\n+\tret = relink_or_rename(tmpfile, filename);\n+\tif (ret)\n+\t\treturn error(\"unable to write sha1 filename %s: %s\",\n+\t\t\t     filename, strerror(ret));\n+\n \tpull_say(\"got %s\\n\", hex);\n \treturn 0;\n }\n@@ -505,6 +641,7 @@ int main(int argc, char **argv)\n \n \tcurl = curl_easy_init();\n \tno_pragma_header = curl_slist_append(no_pragma_header, \"Pragma:\");\n+\tno_range_header = curl_slist_append(no_range_header, \"Range:\");\n \n \tcurl_ssl_verify = getenv(\"GIT_SSL_NO_VERIFY\") ? 0 : 1;\n \tcurl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, curl_ssl_verify);\n"},{"id":"9538","messageId":"7vk6gz6lpp.fsf@assigned-by-dhcp.cox.net","threadId":"1966","inReplyTo":"20050928171404.GA15593@reactrix.com","subject":"Re: [PATCH] HTTP partial transfer support for object, pack, and index transfers","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-30T07:47:30Z","receivedAt":"2005-09-30T07:47:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I took a look at this patch.  It did not cleanly apply anymore,\nbut I merged it anyway, and then took the liberty of updating it\nfurther, according to your response to my earlier comments.\n\nThey will appear near the tip of the proposed updates branch.\nI'd appreciate it if you could check it out and see I did not\nbreak things by mistake.\n"},{"id":"9571","messageId":"20050930232747.GB15593@reactrix.com","threadId":"1966","inReplyTo":"7vk6gz6lpp.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] HTTP partial transfer support for object, pack, and index transfers","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-09-30T23:27:47Z","receivedAt":"2005-09-30T23:27:47Z","isPatch":true,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"On Fri, Sep 30, 2005 at 12:47:30AM -0700, Junio C Hamano wrote:\n\n> I took a look at this patch.  It did not cleanly apply anymore,\n> but I merged it anyway, and then took the liberty of updating it\n> further, according to your response to my earlier comments.\n\nSorry about that - we're actually using git as a back end to a content\ndistribution system, so I'm still learning the finer points of using it\nfor source control...\n\n> They will appear near the tip of the proposed updates branch.\n> I'd appreciate it if you could check it out and see I did not\n> break things by mistake.\n\nI've included a patch that fixes one small problem; I've tested partial\ntransfers of packs and objects with this applied and it looks good.\n\n\n\n\nDon't unlink the temp file when an object transfer fails, so next attempt\nwill pick up where the failed transfer left off\n\nSigned-off-by: Nick Hengeveld <nickh@reactrix.com>\n\n\n---\n\n http-fetch.c |    1 -\n 1 files changed, 0 insertions(+), 1 deletions(-)\n\n22226e509b916958a8b7aae76945c08d15ec686a\ndiff --git a/http-fetch.c b/http-fetch.c\n--- a/http-fetch.c\n+++ b/http-fetch.c\n@@ -550,7 +550,6 @@ static int fetch_object(struct alt_base \n \tcurl_result = curl_easy_perform(curl);\n \tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);\n \tif (curl_result != 0) {\n-\t\tunlink(tmpfile);\n \t\treturn error(\"%s\", curl_errorstr);\n \t}\n \n"},{"id":"9584","messageId":"7vk6gx905a.fsf@assigned-by-dhcp.cox.net","threadId":"1966","inReplyTo":"20050930232747.GB15593@reactrix.com","subject":"Re: [PATCH] HTTP partial transfer support for object, pack, and index transfers","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-01T07:17:21Z","receivedAt":"2005-10-01T07:17:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks.\n"}]}