{"thread":{"id":"19700","subject":"[PATCH 23/23] http*: add helper methods for fetching objects (loose)","startedAt":"2009-06-05T16:06:30Z","lastAt":"2009-06-05T16:06:30Z","messageCount":1,"participants":["Tay Ray Chuan"],"isPatch":true,"patchVersion":1,"patchTotal":23},"messages":[{"id":"115536","messageId":"20090606000630.f9fe77c0.rctay89@gmail.com","threadId":"19700","inReplyTo":null,"subject":"[PATCH 23/23] http*: add helper methods for fetching objects (loose)","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2009-06-05T16:06:30Z","receivedAt":"2009-06-05T16:06:30Z","isPatch":true,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"The code handling the fetching of loose objects in http-push.c and\nhttp-walker.c have been refactored into new methods and a new struct\n(object_http_request) in http.c. They are not meant to be invoked\nelsewhere.\n\nThe new methods in http.c are\n - new_http_object_request\n - process_http_object_request\n - finish_http_object_request\n - abort_http_object_request\n - release_http_object_request\n\nand the new struct is http_object_request.\n\nRANGER_HEADER_SIZE and no_pragma_header is no longer made available\noutside of http.c, since after the above changes, there are no other\ninstances of usage outside of http.c.\n\nRemove members of the transfer_request struct in http-push.c and\nhttp-walker.c, including filename, real_sha1 and zret, as they are used\nno longer used.\n\nMove the methods append_remote_object_url() and get_remote_object_url()\nfrom http-push.c to http.c. Additionally, get_remote_object_url() is no\nlonger defined only when USE_CURL_MULTI is defined, since\nnon-USE_CURL_MULTI code in http.c uses it (namely, in\nnew_http_object_request()).\n\nRefactor code from http-push.c::start_fetch_loose() and\nhttp-walker.c::start_object_fetch_request() that deals with the details\nof coming up with the filename to store the retrieved object, resuming\na previously aborted request, and making a new curl request, into a new\nfunction, new_http_object_request().\n\nRefactor code from http-walker.c::process_object_request() into the\nfunction, process_http_object_request().\n\nRefactor code from http-push.c::finish_request() and\nhttp-walker.c::finish_object_request() into a new function,\nfinish_http_object_request(). It returns the result of the\nmove_temp_to_file() invocation.\n\nAdd a function, release_http_object_request(), which cleans up object\nrequest data. http-push.c and http-walker.c invoke this function\nseparately; http-push.c::release_request() and\nhttp-walker.c::release_object_request() do not invoke this function.\n\nAdd a function, abort_http_object_request(), which unlink()s the object\nfile and invokes release_http_object_request(). Update\nhttp-walker.c::abort_object_request() to use this.\n\nSigned-off-by: Tay Ray Chuan <rctay89@gmail.com>\n---\n http-push.c   |  213 +++------------------------------------------\n http-walker.c |  267 +++++++++------------------------------------------------\n http.c        |  250 +++++++++++++++++++++++++++++++++++++++++++++++++++++-\n http.h        |   37 +++++++-\n 4 files changed, 336 insertions(+), 431 deletions(-)\n\ndiff --git a/http-push.c b/http-push.c\nindex 71d9967..0d83465 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -115,18 +115,10 @@ struct transfer_request\n \tstruct remote_lock *lock;\n \tstruct curl_slist *headers;\n \tstruct buffer buffer;\n-\tchar filename[PATH_MAX];\n-\tchar tmpfile[PATH_MAX];\n-\tint local_fileno;\n \tenum transfer_state state;\n \tCURLcode curl_result;\n \tchar errorstr[CURL_ERROR_SIZE];\n \tlong http_code;\n-\tunsigned char real_sha1[20];\n-\tgit_SHA_CTX c;\n-\tz_stream stream;\n-\tint zret;\n-\tint rename;\n \tvoid *userData;\n \tstruct active_request_slot *slot;\n \tstruct transfer_request *next;\n@@ -232,15 +224,6 @@ static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum d\n \treturn dav_headers;\n }\n\n-static void append_remote_object_url(struct strbuf *buf, const char *url,\n-\t\t\t\t     const char *hex,\n-\t\t\t\t     int only_two_digit_prefix)\n-{\n-\tstrbuf_addf(buf, \"%sobjects/%.*s/\", url, 2, hex);\n-\tif (!only_two_digit_prefix)\n-\t\tstrbuf_addf(buf, \"%s\", hex+2);\n-}\n-\n static void finish_request(struct transfer_request *request);\n static void release_request(struct transfer_request *request);\n\n@@ -254,169 +237,29 @@ static void process_response(void *callback_data)\n\n #ifdef USE_CURL_MULTI\n\n-static char *get_remote_object_url(const char *url, const char *hex,\n-\t\t\t\t   int only_two_digit_prefix)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tappend_remote_object_url(&buf, url, hex, only_two_digit_prefix);\n-\treturn strbuf_detach(&buf, NULL);\n-}\n-\n-static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,\n-\t\t\t       void *data)\n-{\n-\tunsigned char expn[4096];\n-\tsize_t size = eltsize * nmemb;\n-\tint posn = 0;\n-\tstruct transfer_request *request = (struct transfer_request *)data;\n-\tdo {\n-\t\tssize_t retval = xwrite(request->local_fileno,\n-\t\t\t\t\t(char *) ptr + posn, size - posn);\n-\t\tif (retval < 0)\n-\t\t\treturn posn;\n-\t\tposn += retval;\n-\t} while (posn < size);\n-\n-\trequest->stream.avail_in = size;\n-\trequest->stream.next_in = ptr;\n-\tdo {\n-\t\trequest->stream.next_out = expn;\n-\t\trequest->stream.avail_out = sizeof(expn);\n-\t\trequest->zret = git_inflate(&request->stream, Z_SYNC_FLUSH);\n-\t\tgit_SHA1_Update(&request->c, expn,\n-\t\t\t\tsizeof(expn) - request->stream.avail_out);\n-\t} while (request->stream.avail_in && request->zret == Z_OK);\n-\tdata_received++;\n-\treturn size;\n-}\n-\n static void start_fetch_loose(struct transfer_request *request)\n {\n-\tchar *hex = sha1_to_hex(request->obj->sha1);\n-\tchar *filename;\n-\tchar prevfile[PATH_MAX];\n-\tchar *url;\n-\tint prevlocal;\n-\tunsigned char prev_buf[PREV_BUF_SIZE];\n-\tssize_t prev_read = 0;\n-\tlong prev_posn = 0;\n-\tchar range[RANGE_HEADER_SIZE];\n-\tstruct curl_slist *range_header = NULL;\n \tstruct active_request_slot *slot;\n+\tstruct http_object_request *obj_req;\n\n-\tfilename = sha1_file_name(request->obj->sha1);\n-\tsnprintf(request->filename, sizeof(request->filename), \"%s\", filename);\n-\tsnprintf(request->tmpfile, sizeof(request->tmpfile),\n-\t\t \"%s.temp\", filename);\n-\n-\tsnprintf(prevfile, sizeof(prevfile), \"%s.prev\", request->filename);\n-\tunlink_or_warn(prevfile);\n-\trename(request->tmpfile, prevfile);\n-\tunlink_or_warn(request->tmpfile);\n-\n-\tif (request->local_fileno != -1)\n-\t\terror(\"fd leakage in start: %d\", request->local_fileno);\n-\trequest->local_fileno = open(request->tmpfile,\n-\t\t\t\t     O_WRONLY | O_CREAT | O_EXCL, 0666);\n-\t/*\n-\t * This could have failed due to the \"lazy directory creation\";\n-\t * try to mkdir the last path component.\n-\t */\n-\tif (request->local_fileno < 0 && errno == ENOENT) {\n-\t\tchar *dir = strrchr(request->tmpfile, '/');\n-\t\tif (dir) {\n-\t\t\t*dir = 0;\n-\t\t\tmkdir(request->tmpfile, 0777);\n-\t\t\t*dir = '/';\n-\t\t}\n-\t\trequest->local_fileno = open(request->tmpfile,\n-\t\t\t\t\t     O_WRONLY | O_CREAT | O_EXCL, 0666);\n-\t}\n-\n-\tif (request->local_fileno < 0) {\n+\tobj_req = new_http_object_request(repo->url, request->obj->sha1);\n+\tif (obj_req == NULL) {\n \t\trequest->state = ABORTED;\n-\t\terror(\"Couldn't create temporary file %s for %s: %s\",\n-\t\t      request->tmpfile, request->filename, strerror(errno));\n \t\treturn;\n \t}\n\n-\tmemset(&request->stream, 0, sizeof(request->stream));\n-\n-\tgit_inflate_init(&request->stream);\n-\n-\tgit_SHA1_Init(&request->c);\n-\n-\turl = get_remote_object_url(repo->url, hex, 0);\n-\trequest->url = xstrdup(url);\n-\n-\t/*\n-\t * If a previous temp file is present, process what was already\n-\t * fetched.\n-\t */\n-\tprevlocal = open(prevfile, O_RDONLY);\n-\tif (prevlocal != -1) {\n-\t\tdo {\n-\t\t\tprev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);\n-\t\t\tif (prev_read>0) {\n-\t\t\t\tif (fwrite_sha1_file(prev_buf,\n-\t\t\t\t\t\t     1,\n-\t\t\t\t\t\t     prev_read,\n-\t\t\t\t\t\t     request) == prev_read)\n-\t\t\t\t\tprev_posn += prev_read;\n-\t\t\t\telse\n-\t\t\t\t\tprev_read = -1;\n-\t\t\t}\n-\t\t} while (prev_read > 0);\n-\t\tclose(prevlocal);\n-\t}\n-\tunlink_or_warn(prevfile);\n-\n-\t/*\n-\t * Reset inflate/SHA1 if there was an error reading the previous temp\n-\t * file; also rewind to the beginning of the local file.\n-\t */\n-\tif (prev_read == -1) {\n-\t\tmemset(&request->stream, 0, sizeof(request->stream));\n-\t\tgit_inflate_init(&request->stream);\n-\t\tgit_SHA1_Init(&request->c);\n-\t\tif (prev_posn>0) {\n-\t\t\tprev_posn = 0;\n-\t\t\tlseek(request->local_fileno, 0, SEEK_SET);\n-\t\t\tftruncate(request->local_fileno, 0);\n-\t\t}\n-\t}\n-\n-\tslot = get_active_slot();\n+\tslot = obj_req->slot;\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \trequest->slot = slot;\n-\n-\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, request);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, request->errorstr);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n-\n-\t/*\n-\t * If we have successfully processed data from a previous fetch\n-\t * attempt, only fetch the data we don't already have.\n-\t */\n-\tif (prev_posn>0) {\n-\t\tif (push_verbosely)\n-\t\t\tfprintf(stderr,\n-\t\t\t\t\"Resuming fetch of object %s at byte %ld\\n\",\n-\t\t\t\thex, prev_posn);\n-\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n-\t\trange_header = curl_slist_append(range_header, range);\n-\t\tcurl_easy_setopt(slot->curl,\n-\t\t\t\t CURLOPT_HTTPHEADER, range_header);\n-\t}\n+\trequest->userData = obj_req;\n\n \t/* Try to get the request started, abort the request on error */\n \trequest->state = RUN_FETCH_LOOSE;\n \tif (!start_active_slot(slot)) {\n \t\tfprintf(stderr, \"Unable to start GET request\\n\");\n \t\trepo->can_update_info_refs = 0;\n+\t\trelease_http_object_request(obj_req);\n \t\trelease_request(request);\n \t}\n }\n@@ -675,16 +518,14 @@ static void release_request(struct transfer_request *request)\n \t\t\tentry->next = entry->next->next;\n \t}\n\n-\tif (request->local_fileno != -1)\n-\t\tclose(request->local_fileno);\n \tfree(request->url);\n \tfree(request);\n }\n\n static void finish_request(struct transfer_request *request)\n {\n-\tstruct stat st;\n \tstruct http_pack_request *preq;\n+\tstruct http_object_request *obj_req;\n\n \trequest->curl_result = request->slot->curl_result;\n \trequest->http_code = request->slot->http_code;\n@@ -739,39 +580,17 @@ static void finish_request(struct transfer_request *request)\n \t\t\taborted = 1;\n \t\t}\n \t} else if (request->state == RUN_FETCH_LOOSE) {\n-\t\tclose(request->local_fileno);\n-\t\trequest->local_fileno = -1;\n-\n-\t\tif (request->curl_result != CURLE_OK &&\n-\t\t    request->http_code != 416) {\n-\t\t\tif (stat(request->tmpfile, &st) == 0) {\n-\t\t\t\tif (st.st_size == 0)\n-\t\t\t\t\tunlink_or_warn(request->tmpfile);\n-\t\t\t}\n-\t\t} else {\n-\t\t\tif (request->http_code == 416)\n-\t\t\t\twarning(\"requested range invalid; we may already have all the data.\");\n-\n-\t\t\tgit_inflate_end(&request->stream);\n-\t\t\tgit_SHA1_Final(request->real_sha1, &request->c);\n-\t\t\tif (request->zret != Z_STREAM_END) {\n-\t\t\t\tunlink_or_warn(request->tmpfile);\n-\t\t\t} else if (hashcmp(request->obj->sha1, request->real_sha1)) {\n-\t\t\t\tunlink_or_warn(request->tmpfile);\n-\t\t\t} else {\n-\t\t\t\trequest->rename =\n-\t\t\t\t\tmove_temp_to_file(\n-\t\t\t\t\t\trequest->tmpfile,\n-\t\t\t\t\t\trequest->filename);\n-\t\t\t\tif (request->rename == 0)\n-\t\t\t\t\trequest->obj->flags |= (LOCAL | REMOTE);\n-\t\t\t}\n-\t\t}\n+\t\tobj_req = (struct http_object_request *)request->userData;\n+\n+\t\tif (finish_http_object_request(obj_req) == 0)\n+\t\t\tif (obj_req->rename == 0)\n+\t\t\t\trequest->obj->flags |= (LOCAL | REMOTE);\n\n \t\t/* Try fetching packed if necessary */\n-\t\tif (request->obj->flags & LOCAL)\n+\t\tif (request->obj->flags & LOCAL) {\n+\t\t\trelease_http_object_request(obj_req);\n \t\t\trelease_request(request);\n-\t\telse\n+\t\t} else\n \t\t\tstart_fetch_packed(request);\n\n \t} else if (request->state == RUN_FETCH_PACKED) {\n@@ -843,7 +662,6 @@ static void add_fetch_request(struct object *obj)\n \trequest->url = NULL;\n \trequest->lock = NULL;\n \trequest->headers = NULL;\n-\trequest->local_fileno = -1;\n \trequest->state = NEED_FETCH;\n \trequest->next = request_queue_head;\n \trequest_queue_head = request;\n@@ -882,7 +700,6 @@ static int add_send_request(struct object *obj, struct remote_lock *lock)\n \trequest->url = NULL;\n \trequest->lock = lock;\n \trequest->headers = NULL;\n-\trequest->local_fileno = -1;\n \trequest->state = NEED_PUSH;\n \trequest->next = request_queue_head;\n \trequest_queue_head = request;\ndiff --git a/http-walker.c b/http-walker.c\nindex 8f7a975..700bc13 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -3,8 +3,6 @@\n #include \"walker.h\"\n #include \"http.h\"\n\n-#define PREV_BUF_SIZE 4096\n-\n struct alt_base\n {\n \tchar *base;\n@@ -25,20 +23,8 @@ struct object_request\n \tstruct walker *walker;\n \tunsigned char sha1[20];\n \tstruct alt_base *repo;\n-\tchar *url;\n-\tchar filename[PATH_MAX];\n-\tchar tmpfile[PATH_MAX];\n-\tint local;\n \tenum object_request_state state;\n-\tCURLcode curl_result;\n-\tchar errorstr[CURL_ERROR_SIZE];\n-\tlong http_code;\n-\tunsigned char real_sha1[20];\n-\tgit_SHA_CTX c;\n-\tz_stream stream;\n-\tint zret;\n-\tint rename;\n-\tstruct active_request_slot *slot;\n+\tstruct http_object_request *req;\n \tstruct object_request *next;\n };\n\n@@ -59,34 +45,6 @@ struct walker_data {\n\n static struct object_request *object_queue_head;\n\n-static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,\n-\t\t\t       void *data)\n-{\n-\tunsigned char expn[4096];\n-\tsize_t size = eltsize * nmemb;\n-\tint posn = 0;\n-\tstruct object_request *obj_req = (struct object_request *)data;\n-\tdo {\n-\t\tssize_t retval = xwrite(obj_req->local,\n-\t\t\t\t\t(char *) ptr + posn, size - posn);\n-\t\tif (retval < 0)\n-\t\t\treturn posn;\n-\t\tposn += retval;\n-\t} while (posn < size);\n-\n-\tobj_req->stream.avail_in = size;\n-\tobj_req->stream.next_in = ptr;\n-\tdo {\n-\t\tobj_req->stream.next_out = expn;\n-\t\tobj_req->stream.avail_out = sizeof(expn);\n-\t\tobj_req->zret = git_inflate(&obj_req->stream, Z_SYNC_FLUSH);\n-\t\tgit_SHA1_Update(&obj_req->c, expn,\n-\t\t\t\tsizeof(expn) - obj_req->stream.avail_out);\n-\t} while (obj_req->stream.avail_in && obj_req->zret == Z_OK);\n-\tdata_received++;\n-\treturn size;\n-}\n-\n static void fetch_alternates(struct walker *walker, const char *base);\n\n static void process_object_response(void *callback_data);\n@@ -94,172 +52,35 @@ static void process_object_response(void *callback_data);\n static void start_object_request(struct walker *walker,\n \t\t\t\t struct object_request *obj_req)\n {\n-\tchar *hex = sha1_to_hex(obj_req->sha1);\n-\tchar prevfile[PATH_MAX];\n-\tchar *url;\n-\tchar *posn;\n-\tint prevlocal;\n-\tunsigned char prev_buf[PREV_BUF_SIZE];\n-\tssize_t prev_read = 0;\n-\tlong prev_posn = 0;\n-\tchar range[RANGE_HEADER_SIZE];\n-\tstruct curl_slist *range_header = NULL;\n \tstruct active_request_slot *slot;\n+\tstruct http_object_request *req;\n\n-\tsnprintf(prevfile, sizeof(prevfile), \"%s.prev\", obj_req->filename);\n-\tunlink_or_warn(prevfile);\n-\trename(obj_req->tmpfile, prevfile);\n-\tunlink_or_warn(obj_req->tmpfile);\n-\n-\tif (obj_req->local != -1)\n-\t\terror(\"fd leakage in start: %d\", obj_req->local);\n-\tobj_req->local = open(obj_req->tmpfile,\n-\t\t\t      O_WRONLY | O_CREAT | O_EXCL, 0666);\n-\t/*\n-\t * This could have failed due to the \"lazy directory creation\";\n-\t * try to mkdir the last path component.\n-\t */\n-\tif (obj_req->local < 0 && errno == ENOENT) {\n-\t\tchar *dir = strrchr(obj_req->tmpfile, '/');\n-\t\tif (dir) {\n-\t\t\t*dir = 0;\n-\t\t\tmkdir(obj_req->tmpfile, 0777);\n-\t\t\t*dir = '/';\n-\t\t}\n-\t\tobj_req->local = open(obj_req->tmpfile,\n-\t\t\t\t      O_WRONLY | O_CREAT | O_EXCL, 0666);\n-\t}\n-\n-\tif (obj_req->local < 0) {\n+\treq = new_http_object_request(obj_req->repo->base, obj_req->sha1);\n+\tif (req == NULL) {\n \t\tobj_req->state = ABORTED;\n-\t\terror(\"Couldn't create temporary file %s for %s: %s\",\n-\t\t      obj_req->tmpfile, obj_req->filename, strerror(errno));\n \t\treturn;\n \t}\n+\tobj_req->req = req;\n\n-\tmemset(&obj_req->stream, 0, sizeof(obj_req->stream));\n-\n-\tgit_inflate_init(&obj_req->stream);\n-\n-\tgit_SHA1_Init(&obj_req->c);\n-\n-\turl = xmalloc(strlen(obj_req->repo->base) + 51);\n-\tobj_req->url = xmalloc(strlen(obj_req->repo->base) + 51);\n-\tstrcpy(url, obj_req->repo->base);\n-\tposn = url + strlen(obj_req->repo->base);\n-\tstrcpy(posn, \"/objects/\");\n-\tposn += 9;\n-\tmemcpy(posn, hex, 2);\n-\tposn += 2;\n-\t*(posn++) = '/';\n-\tstrcpy(posn, hex + 2);\n-\tstrcpy(obj_req->url, url);\n-\n-\t/*\n-\t * If a previous temp file is present, process what was already\n-\t * fetched.\n-\t */\n-\tprevlocal = open(prevfile, O_RDONLY);\n-\tif (prevlocal != -1) {\n-\t\tdo {\n-\t\t\tprev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);\n-\t\t\tif (prev_read>0) {\n-\t\t\t\tif (fwrite_sha1_file(prev_buf,\n-\t\t\t\t\t\t     1,\n-\t\t\t\t\t\t     prev_read,\n-\t\t\t\t\t\t     obj_req) == prev_read)\n-\t\t\t\t\tprev_posn += prev_read;\n-\t\t\t\telse\n-\t\t\t\t\tprev_read = -1;\n-\t\t\t}\n-\t\t} while (prev_read > 0);\n-\t\tclose(prevlocal);\n-\t}\n-\tunlink_or_warn(prevfile);\n-\n-\t/*\n-\t * Reset inflate/SHA1 if there was an error reading the previous temp\n-\t * file; also rewind to the beginning of the local file.\n-\t */\n-\tif (prev_read == -1) {\n-\t\tmemset(&obj_req->stream, 0, sizeof(obj_req->stream));\n-\t\tgit_inflate_init(&obj_req->stream);\n-\t\tgit_SHA1_Init(&obj_req->c);\n-\t\tif (prev_posn>0) {\n-\t\t\tprev_posn = 0;\n-\t\t\tlseek(obj_req->local, 0, SEEK_SET);\n-\t\t\tftruncate(obj_req->local, 0);\n-\t\t}\n-\t}\n-\n-\tslot = get_active_slot();\n+\tslot = req->slot;\n \tslot->callback_func = process_object_response;\n \tslot->callback_data = obj_req;\n-\tobj_req->slot = slot;\n-\n-\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, obj_req);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, obj_req->errorstr);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n-\n-\t/*\n-\t * If we have successfully processed data from a previous fetch\n-\t * attempt, only fetch the data we don't already have.\n-\t */\n-\tif (prev_posn>0) {\n-\t\tif (walker->get_verbosely)\n-\t\t\tfprintf(stderr,\n-\t\t\t\t\"Resuming fetch of object %s at byte %ld\\n\",\n-\t\t\t\thex, prev_posn);\n-\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n-\t\trange_header = curl_slist_append(range_header, range);\n-\t\tcurl_easy_setopt(slot->curl,\n-\t\t\t\t CURLOPT_HTTPHEADER, range_header);\n-\t}\n\n \t/* Try to get the request started, abort the request on error */\n \tobj_req->state = ACTIVE;\n \tif (!start_active_slot(slot)) {\n \t\tobj_req->state = ABORTED;\n-\t\tobj_req->slot = NULL;\n-\t\tclose(obj_req->local);\n-\t\tobj_req->local = -1;\n-\t\tfree(obj_req->url);\n+\t\trelease_http_object_request(req);\n \t\treturn;\n \t}\n }\n\n static void finish_object_request(struct object_request *obj_req)\n {\n-\tstruct stat st;\n-\n-\tclose(obj_req->local);\n-\tobj_req->local = -1;\n-\n-\tif (obj_req->http_code == 416) {\n-\t\tfprintf(stderr, \"Warning: requested range invalid; we may already have all the data.\\n\");\n-\t} else if (obj_req->curl_result != CURLE_OK) {\n-\t\tif (stat(obj_req->tmpfile, &st) == 0)\n-\t\t\tif (st.st_size == 0)\n-\t\t\t\tunlink_or_warn(obj_req->tmpfile);\n+\tif (finish_http_object_request(obj_req->req))\n \t\treturn;\n-\t}\n-\n-\tgit_inflate_end(&obj_req->stream);\n-\tgit_SHA1_Final(obj_req->real_sha1, &obj_req->c);\n-\tif (obj_req->zret != Z_STREAM_END) {\n-\t\tunlink_or_warn(obj_req->tmpfile);\n-\t\treturn;\n-\t}\n-\tif (hashcmp(obj_req->sha1, obj_req->real_sha1)) {\n-\t\tunlink_or_warn(obj_req->tmpfile);\n-\t\treturn;\n-\t}\n-\tobj_req->rename =\n-\t\tmove_temp_to_file(obj_req->tmpfile, obj_req->filename);\n\n-\tif (obj_req->rename == 0)\n+\tif (obj_req->req->rename == 0)\n \t\twalker_say(obj_req->walker, \"got %s\\n\", sha1_to_hex(obj_req->sha1));\n }\n\n@@ -271,19 +92,16 @@ static void process_object_response(void *callback_data)\n \tstruct walker_data *data = walker->data;\n \tstruct alt_base *alt = data->alt;\n\n-\tobj_req->curl_result = obj_req->slot->curl_result;\n-\tobj_req->http_code = obj_req->slot->http_code;\n-\tobj_req->slot = NULL;\n+\tprocess_http_object_request(obj_req->req);\n \tobj_req->state = COMPLETE;\n\n \t/* Use alternates if necessary */\n-\tif (missing_target(obj_req)) {\n+\tif (missing_target(obj_req->req)) {\n \t\tfetch_alternates(walker, alt->base);\n \t\tif (obj_req->repo->next != NULL) {\n \t\t\tobj_req->repo =\n \t\t\t\tobj_req->repo->next;\n-\t\t\tclose(obj_req->local);\n-\t\t\tobj_req->local = -1;\n+\t\t\trelease_http_object_request(obj_req->req);\n \t\t\tstart_object_request(walker, obj_req);\n \t\t\treturn;\n \t\t}\n@@ -296,8 +114,8 @@ static void release_object_request(struct object_request *obj_req)\n {\n \tstruct object_request *entry = object_queue_head;\n\n-\tif (obj_req->local != -1)\n-\t\terror(\"fd leakage in release: %d\", obj_req->local);\n+\tif (obj_req->req !=NULL && obj_req->req->localfile != -1)\n+\t\terror(\"fd leakage in release: %d\", obj_req->req->localfile);\n \tif (obj_req == object_queue_head) {\n \t\tobject_queue_head = obj_req->next;\n \t} else {\n@@ -307,7 +125,6 @@ static void release_object_request(struct object_request *obj_req)\n \t\t\tentry->next = entry->next->next;\n \t}\n\n-\tfree(obj_req->url);\n \tfree(obj_req);\n }\n\n@@ -335,19 +152,13 @@ static void prefetch(struct walker *walker, unsigned char *sha1)\n \tstruct object_request *newreq;\n \tstruct object_request *tail;\n \tstruct walker_data *data = walker->data;\n-\tchar *filename = sha1_file_name(sha1);\n\n \tnewreq = xmalloc(sizeof(*newreq));\n \tnewreq->walker = walker;\n \thashcpy(newreq->sha1, sha1);\n \tnewreq->repo = data->alt;\n-\tnewreq->url = NULL;\n-\tnewreq->local = -1;\n \tnewreq->state = WAITING;\n-\tsnprintf(newreq->filename, sizeof(newreq->filename), \"%s\", filename);\n-\tsnprintf(newreq->tmpfile, sizeof(newreq->tmpfile),\n-\t\t \"%s.temp\", filename);\n-\tnewreq->slot = NULL;\n+\tnewreq->req = NULL;\n \tnewreq->next = NULL;\n\n \thttp_is_verbose = walker->get_verbosely;\n@@ -638,15 +449,6 @@ abort:\n\n static void abort_object_request(struct object_request *obj_req)\n {\n-\tif (obj_req->local >= 0) {\n-\t\tclose(obj_req->local);\n-\t\tobj_req->local = -1;\n-\t}\n-\tunlink_or_warn(obj_req->tmpfile);\n-\tif (obj_req->slot) {\n-\t\trelease_active_slot(obj_req->slot);\n-\t\tobj_req->slot = NULL;\n-\t}\n \trelease_object_request(obj_req);\n }\n\n@@ -655,6 +457,7 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c\n \tchar *hex = sha1_to_hex(sha1);\n \tint ret = 0;\n \tstruct object_request *obj_req = object_queue_head;\n+\tstruct http_object_request *req;\n\n \twhile (obj_req != NULL && hashcmp(obj_req->sha1, sha1))\n \t\tobj_req = obj_req->next;\n@@ -662,6 +465,8 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c\n \t\treturn error(\"Couldn't find request for %s in the queue\", hex);\n\n \tif (has_sha1_file(obj_req->sha1)) {\n+\t\tif (obj_req->req != NULL)\n+\t\t\tabort_http_object_request(obj_req->req);\n \t\tabort_object_request(obj_req);\n \t\treturn 0;\n \t}\n@@ -673,34 +478,42 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c\n \tstart_object_request(walker, obj_req);\n #endif\n\n+\t/*\n+\t * obj_req->req might change when fetching alternates in the callback\n+\t * process_object_response; therefore, the \"shortcut\" variable, req,\n+\t * is used only after we're done with slots.\n+\t */\n \twhile (obj_req->state == ACTIVE)\n-\t\trun_active_slot(obj_req->slot);\n+\t\trun_active_slot(obj_req->req->slot);\n+\n+\treq = obj_req->req;\n\n-\tif (obj_req->local != -1) {\n-\t\tclose(obj_req->local);\n-\t\tobj_req->local = -1;\n+\tif (req->localfile != -1) {\n+\t\tclose(req->localfile);\n+\t\treq->localfile = -1;\n \t}\n\n \tif (obj_req->state == ABORTED) {\n \t\tret = error(\"Request for %s aborted\", hex);\n-\t} else if (obj_req->curl_result != CURLE_OK &&\n-\t\t   obj_req->http_code != 416) {\n-\t\tif (missing_target(obj_req))\n+\t} else if (req->curl_result != CURLE_OK &&\n+\t\t   req->http_code != 416) {\n+\t\tif (missing_target(req))\n \t\t\tret = -1; /* Be silent, it is probably in a pack. */\n \t\telse\n \t\t\tret = error(\"%s (curl_result = %d, http_code = %ld, sha1 = %s)\",\n-\t\t\t\t    obj_req->errorstr, obj_req->curl_result,\n-\t\t\t\t    obj_req->http_code, hex);\n-\t} else if (obj_req->zret != Z_STREAM_END) {\n+\t\t\t\t    req->errorstr, req->curl_result,\n+\t\t\t\t    req->http_code, hex);\n+\t} else if (req->zret != Z_STREAM_END) {\n \t\twalker->corrupt_object_found++;\n-\t\tret = error(\"File %s (%s) corrupt\", hex, obj_req->url);\n-\t} else if (hashcmp(obj_req->sha1, obj_req->real_sha1)) {\n+\t\tret = error(\"File %s (%s) corrupt\", hex, req->url);\n+\t} else if (hashcmp(obj_req->sha1, req->real_sha1)) {\n \t\tret = error(\"File %s has bad hash\", hex);\n-\t} else if (obj_req->rename < 0) {\n+\t} else if (req->rename < 0) {\n \t\tret = error(\"unable to write sha1 filename %s\",\n-\t\t\t    obj_req->filename);\n+\t\t\t    req->filename);\n \t}\n\n+\trelease_http_object_request(req);\n \trelease_object_request(obj_req);\n \treturn ret;\n }\ndiff --git a/http.c b/http.c\nindex 3815934..95b2137 100644\n--- a/http.c\n+++ b/http.c\n@@ -12,6 +12,10 @@ static CURLM *curlm;\n #ifndef NO_CURL_EASY_DUPHANDLE\n static CURL *curl_default;\n #endif\n+\n+#define PREV_BUF_SIZE 4096\n+#define RANGE_HEADER_SIZE 30\n+\n char curl_errorstr[CURL_ERROR_SIZE];\n\n static int curl_ssl_verify = -1;\n@@ -30,8 +34,7 @@ static const char *curl_http_proxy;\n static char *user_name, *user_pass;\n\n static struct curl_slist *pragma_header;\n-\n-struct curl_slist *no_pragma_header;\n+static struct curl_slist *no_pragma_header;\n\n static struct active_request_slot *active_queue_head;\n\n@@ -666,6 +669,23 @@ static char *quote_ref_url(const char *base, const char *ref)\n \treturn strbuf_detach(&buf, NULL);\n }\n\n+void append_remote_object_url(struct strbuf *buf, const char *url,\n+\t\t\t      const char *hex,\n+\t\t\t      int only_two_digit_prefix)\n+{\n+\tstrbuf_addf(buf, \"%s/objects/%.*s/\", url, 2, hex);\n+\tif (!only_two_digit_prefix)\n+\t\tstrbuf_addf(buf, \"%s\", hex+2);\n+}\n+\n+char *get_remote_object_url(const char *url, const char *hex,\n+\t\t\t    int only_two_digit_prefix)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tappend_remote_object_url(&buf, url, hex, only_two_digit_prefix);\n+\treturn strbuf_detach(&buf, NULL);\n+}\n+\n /* http_request() targets */\n #define HTTP_REQUEST_STRBUF\t0\n #define HTTP_REQUEST_FILE\t1\n@@ -995,3 +1015,229 @@ abort:\n \tfree(filename);\n \treturn NULL;\n }\n+\n+/* Helpers for fetching objects (loose) */\n+static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,\n+\t\t\t       void *data)\n+{\n+\tunsigned char expn[4096];\n+\tsize_t size = eltsize * nmemb;\n+\tint posn = 0;\n+\tstruct http_object_request *freq =\n+\t\t(struct http_object_request *)data;\n+\tdo {\n+\t\tssize_t retval = xwrite(freq->localfile,\n+\t\t\t\t\t(char *) ptr + posn, size - posn);\n+\t\tif (retval < 0)\n+\t\t\treturn posn;\n+\t\tposn += retval;\n+\t} while (posn < size);\n+\n+\tfreq->stream.avail_in = size;\n+\tfreq->stream.next_in = ptr;\n+\tdo {\n+\t\tfreq->stream.next_out = expn;\n+\t\tfreq->stream.avail_out = sizeof(expn);\n+\t\tfreq->zret = git_inflate(&freq->stream, Z_SYNC_FLUSH);\n+\t\tgit_SHA1_Update(&freq->c, expn,\n+\t\t\t\tsizeof(expn) - freq->stream.avail_out);\n+\t} while (freq->stream.avail_in && freq->zret == Z_OK);\n+\tdata_received++;\n+\treturn size;\n+}\n+\n+struct http_object_request *new_http_object_request(const char *base_url,\n+\tunsigned char *sha1)\n+{\n+\tchar *hex = sha1_to_hex(sha1);\n+\tchar *filename;\n+\tchar prevfile[PATH_MAX];\n+\tchar *url;\n+\tint prevlocal;\n+\tunsigned char prev_buf[PREV_BUF_SIZE];\n+\tssize_t prev_read = 0;\n+\tlong prev_posn = 0;\n+\tchar range[RANGE_HEADER_SIZE];\n+\tstruct curl_slist *range_header = NULL;\n+\tstruct http_object_request *freq;\n+\n+\tfreq = xmalloc(sizeof(*freq));\n+\thashcpy(freq->sha1, sha1);\n+\tfreq->localfile = -1;\n+\n+\tfilename = sha1_file_name(sha1);\n+\tsnprintf(freq->filename, sizeof(freq->filename), \"%s\", filename);\n+\tsnprintf(freq->tmpfile, sizeof(freq->tmpfile),\n+\t\t \"%s.temp\", filename);\n+\n+\tsnprintf(prevfile, sizeof(prevfile), \"%s.prev\", filename);\n+\tunlink_or_warn(prevfile);\n+\trename(freq->tmpfile, prevfile);\n+\tunlink_or_warn(freq->tmpfile);\n+\n+\tif (freq->localfile != -1)\n+\t\terror(\"fd leakage in start: %d\", freq->localfile);\n+\tfreq->localfile = open(freq->tmpfile,\n+\t\t\t       O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\t/*\n+\t * This could have failed due to the \"lazy directory creation\";\n+\t * try to mkdir the last path component.\n+\t */\n+\tif (freq->localfile < 0 && errno == ENOENT) {\n+\t\tchar *dir = strrchr(freq->tmpfile, '/');\n+\t\tif (dir) {\n+\t\t\t*dir = 0;\n+\t\t\tmkdir(freq->tmpfile, 0777);\n+\t\t\t*dir = '/';\n+\t\t}\n+\t\tfreq->localfile = open(freq->tmpfile,\n+\t\t\t\t       O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\t}\n+\n+\tif (freq->localfile < 0) {\n+\t\terror(\"Couldn't create temporary file %s for %s: %s\",\n+\t\t      freq->tmpfile, freq->filename, strerror(errno));\n+\t\tgoto abort;\n+\t}\n+\n+\tmemset(&freq->stream, 0, sizeof(freq->stream));\n+\n+\tgit_inflate_init(&freq->stream);\n+\n+\tgit_SHA1_Init(&freq->c);\n+\n+\turl = get_remote_object_url(base_url, hex, 0);\n+\tfreq->url = xstrdup(url);\n+\n+\t/*\n+\t * If a previous temp file is present, process what was already\n+\t * fetched.\n+\t */\n+\tprevlocal = open(prevfile, O_RDONLY);\n+\tif (prevlocal != -1) {\n+\t\tdo {\n+\t\t\tprev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);\n+\t\t\tif (prev_read>0) {\n+\t\t\t\tif (fwrite_sha1_file(prev_buf,\n+\t\t\t\t\t\t     1,\n+\t\t\t\t\t\t     prev_read,\n+\t\t\t\t\t\t     freq) == prev_read) {\n+\t\t\t\t\tprev_posn += prev_read;\n+\t\t\t\t} else {\n+\t\t\t\t\tprev_read = -1;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t} while (prev_read > 0);\n+\t\tclose(prevlocal);\n+\t}\n+\tunlink_or_warn(prevfile);\n+\n+\t/*\n+\t * Reset inflate/SHA1 if there was an error reading the previous temp\n+\t * file; also rewind to the beginning of the local file.\n+\t */\n+\tif (prev_read == -1) {\n+\t\tmemset(&freq->stream, 0, sizeof(freq->stream));\n+\t\tgit_inflate_init(&freq->stream);\n+\t\tgit_SHA1_Init(&freq->c);\n+\t\tif (prev_posn>0) {\n+\t\t\tprev_posn = 0;\n+\t\t\tlseek(freq->localfile, 0, SEEK_SET);\n+\t\t\tftruncate(freq->localfile, 0);\n+\t\t}\n+\t}\n+\n+\tfreq->slot = get_active_slot();\n+\n+\tcurl_easy_setopt(freq->slot->curl, CURLOPT_FILE, freq);\n+\tcurl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n+\tcurl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);\n+\tcurl_easy_setopt(freq->slot->curl, CURLOPT_URL, url);\n+\tcurl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n+\n+\t/*\n+\t * If we have successfully processed data from a previous fetch\n+\t * attempt, only fetch the data we don't already have.\n+\t */\n+\tif (prev_posn>0) {\n+\t\tif (http_is_verbose)\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"Resuming fetch of object %s at byte %ld\\n\",\n+\t\t\t\thex, prev_posn);\n+\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n+\t\trange_header = curl_slist_append(range_header, range);\n+\t\tcurl_easy_setopt(freq->slot->curl,\n+\t\t\t\t CURLOPT_HTTPHEADER, range_header);\n+\t}\n+\n+\treturn freq;\n+\n+\tfree(url);\n+abort:\n+\tfree(filename);\n+\tfree(freq);\n+\treturn NULL;\n+}\n+\n+void process_http_object_request(struct http_object_request *freq)\n+{\n+\tif (freq->slot == NULL)\n+\t\treturn;\n+\tfreq->curl_result = freq->slot->curl_result;\n+\tfreq->http_code = freq->slot->http_code;\n+\tfreq->slot = NULL;\n+}\n+\n+int finish_http_object_request(struct http_object_request *freq)\n+{\n+\tstruct stat st;\n+\n+\tclose(freq->localfile);\n+\tfreq->localfile = -1;\n+\n+\tprocess_http_object_request(freq);\n+\n+\tif (freq->http_code == 416) {\n+\t\tfprintf(stderr, \"Warning: requested range invalid; we may already have all the data.\\n\");\n+\t} else if (freq->curl_result != CURLE_OK) {\n+\t\tif (stat(freq->tmpfile, &st) == 0)\n+\t\t\tif (st.st_size == 0)\n+\t\t\t\tunlink_or_warn(freq->tmpfile);\n+\t\treturn -1;\n+\t}\n+\n+\tgit_inflate_end(&freq->stream);\n+\tgit_SHA1_Final(freq->real_sha1, &freq->c);\n+\tif (freq->zret != Z_STREAM_END) {\n+\t\tunlink_or_warn(freq->tmpfile);\n+\t\treturn -1;\n+\t}\n+\tif (hashcmp(freq->sha1, freq->real_sha1)) {\n+\t\tunlink_or_warn(freq->tmpfile);\n+\t\treturn -1;\n+\t}\n+\tfreq->rename =\n+\t\tmove_temp_to_file(freq->tmpfile, freq->filename);\n+\n+\treturn freq->rename;\n+}\n+\n+void abort_http_object_request(struct http_object_request *freq)\n+{\n+\tunlink_or_warn(freq->tmpfile);\n+\n+\trelease_http_object_request(freq);\n+}\n+\n+void release_http_object_request(struct http_object_request *freq)\n+{\n+\tif (freq->localfile != -1) {\n+\t\tclose(freq->localfile);\n+\t\tfreq->localfile = -1;\n+\t}\n+\tif (freq->url != NULL) {\n+\t\tfree(freq->url);\n+\t\tfreq->url = NULL;\n+\t}\n+\tfreq->slot = NULL;\n+}\ndiff --git a/http.h b/http.h\nindex 511c0c4..4c4e99c 100644\n--- a/http.h\n+++ b/http.h\n@@ -88,10 +88,6 @@ extern void add_fill_function(void *data, int (*fill)(void *));\n extern void step_active_slots(void);\n #endif\n\n-extern struct curl_slist *no_pragma_header;\n-\n-#define RANGE_HEADER_SIZE 30\n-\n extern void http_init(struct remote *remote);\n extern void http_cleanup(void);\n\n@@ -114,6 +110,13 @@ static inline int missing__target(int code, int result)\n\n #define missing_target(a) missing__target((a)->http_code, (a)->curl_result)\n\n+/* Helpers for modifying and creating URLs */\n+extern void append_remote_object_url(struct strbuf *buf, const char *url,\n+\t\t\t\t     const char *hex,\n+\t\t\t\t     int only_two_digit_prefix);\n+extern char *get_remote_object_url(const char *url, const char *hex,\n+\t\t\t\t   int only_two_digit_prefix);\n+\n /* Options for http_request_*() */\n #define HTTP_NO_CACHE\t\t1\n\n@@ -167,4 +170,30 @@ extern struct http_pack_request *new_http_pack_request(\n extern int finish_http_pack_request(struct http_pack_request *preq);\n extern void release_http_pack_request(struct http_pack_request *preq);\n\n+/* Helpers for fetching object */\n+struct http_object_request\n+{\n+\tchar *url;\n+\tchar filename[PATH_MAX];\n+\tchar tmpfile[PATH_MAX];\n+\tint localfile;\n+\tCURLcode curl_result;\n+\tchar errorstr[CURL_ERROR_SIZE];\n+\tlong http_code;\n+\tunsigned char sha1[20];\n+\tunsigned char real_sha1[20];\n+\tgit_SHA_CTX c;\n+\tz_stream stream;\n+\tint zret;\n+\tint rename;\n+\tstruct active_request_slot *slot;\n+};\n+\n+extern struct http_object_request *new_http_object_request(\n+\tconst char *base_url, unsigned char *sha1);\n+extern void process_http_object_request(struct http_object_request *freq);\n+extern int finish_http_object_request(struct http_object_request *freq);\n+extern void abort_http_object_request(struct http_object_request *freq);\n+extern void release_http_object_request(struct http_object_request *freq);\n+\n #endif /* HTTP_H */\n--\n1.6.3.1\n"}]}