{"thread":{"id":"2025","subject":"[PATCH] Add support for parallel HTTP transfers","startedAt":"2005-10-05T21:44:47Z","lastAt":"2005-10-10T16:48:42Z","messageCount":14,"participants":["Nick Hengeveld","Daniel Barkalow","Junio C Hamano","Jon Loeliger"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"9728","messageId":"20051005214447.GF15593@reactrix.com","threadId":"2025","inReplyTo":null,"subject":"[PATCH] Add support for parallel HTTP transfers","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-10-05T21:44:47Z","receivedAt":"2005-10-05T21:44:47Z","isPatch":true,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"Add support for parallel HTTP transfers.  Prefetch populates a queue of\nobjects to transfer and starts feeding requests to an active request\nqueue for processing; fetch_object keeps the active queue moving\nwhile the specified object is being transferred.  The size of the active\nqueue can be restricted using -r and defaults to 5 concurrent transfers.\n\nSigned-off-by: Nick Hengeveld <nickh@reactrix.com>\n\n\n---\n\nI could use extra eyes on this patch - it seems to be stable although I have\nseen periodic cases during testing where it detects an empty server response\n(from kernel.org?)  Are there implications to downloading a (potentially large)\npack while objects contained in that pack have been prefetched and are in the\ntransfer and/or active queue?\n\n\n http-fetch.c |  696 +++++++++++++++++++++++++++++++++++++++++-----------------\n 1 files changed, 494 insertions(+), 202 deletions(-)\n\nfe069497b0f959d4f270e4f01000480dd50febb2\ndiff --git a/http-fetch.c b/http-fetch.c\n--- a/http-fetch.c\n+++ b/http-fetch.c\n@@ -6,6 +6,8 @@\n #include <curl/curl.h>\n #include <curl/easy.h>\n \n+#define DEFAULT_MAX_REQUESTS 5\n+\n #if LIBCURL_VERSION_NUM < 0x070704\n #define curl_global_cleanup() do { /* nothing */ } while(0)\n #endif\n@@ -16,13 +18,14 @@\n #define PREV_BUF_SIZE 4096\n #define RANGE_HEADER_SIZE 30\n \n-static CURL *curl;\n+static int max_requests = DEFAULT_MAX_REQUESTS;\n+\n+static CURLM *curlm;\n+static CURL *curl_default;\n static struct curl_slist *no_pragma_header;\n static struct curl_slist *no_range_header;\n static char curl_errorstr[CURL_ERROR_SIZE];\n \n-static char *initial_base;\n-\n struct alt_base\n {\n \tchar *base;\n@@ -33,11 +36,43 @@ struct alt_base\n \n static struct alt_base *alt = NULL;\n \n-static SHA_CTX c;\n-static z_stream stream;\n+enum transfer_state {\n+\tWAITING,\n+\tABORTED,\n+\tACTIVE,\n+\tCOMPLETE,\n+};\n+\n+struct transfer_request\n+{\n+\tunsigned char sha1[20];\n+\tstruct alt_base *repo;\n+\tchar *url;\n+\tchar filename[PATH_MAX];\n+\tchar tmpfile[PATH_MAX];\n+\tint local;\n+\tenum transfer_state state;\n+\tCURLcode curl_result;\n+\tchar errorstr[CURL_ERROR_SIZE];\n+\tlong http_code;\n+\tunsigned char real_sha1[20];\n+\tSHA_CTX c;\n+\tz_stream stream;\n+\tint zret;\n+\tint rename;\n+\tstruct active_request_slot *slot;\n+\tstruct transfer_request *next;\n+};\n+\n+struct active_request_slot\n+{\n+\tCURL *curl;\n+\tint in_use;\n+\tstruct active_request_slot *next;\n+};\n \n-static int local;\n-static int zret;\n+static struct transfer_request *request_queue_head = NULL;\n+static struct active_request_slot *active_queue_head = NULL;\n \n static int curl_ssl_verify;\n static char *ssl_cert;\n@@ -69,28 +104,27 @@ static size_t fwrite_sha1_file(void *ptr\n \tunsigned char expn[4096];\n \tsize_t size = eltsize * nmemb;\n \tint posn = 0;\n+\tstruct transfer_request *request = (struct transfer_request *)data;\n \tdo {\n-\t\tssize_t retval = write(local, ptr + posn, size - posn);\n+\t\tssize_t retval = write(request->local,\n+\t\t\t\t       ptr + posn, size - posn);\n \t\tif (retval < 0)\n \t\t\treturn posn;\n \t\tposn += retval;\n \t} while (posn < size);\n \n-\tstream.avail_in = size;\n-\tstream.next_in = ptr;\n+\trequest->stream.avail_in = size;\n+\trequest->stream.next_in = ptr;\n \tdo {\n-\t\tstream.next_out = expn;\n-\t\tstream.avail_out = sizeof(expn);\n-\t\tzret = inflate(&stream, Z_SYNC_FLUSH);\n-\t\tSHA1_Update(&c, expn, sizeof(expn) - stream.avail_out);\n-\t} while (stream.avail_in && zret == Z_OK);\n+\t\trequest->stream.next_out = expn;\n+\t\trequest->stream.avail_out = sizeof(expn);\n+\t\trequest->zret = inflate(&request->stream, Z_SYNC_FLUSH);\n+\t\tSHA1_Update(&request->c, expn,\n+\t\t\t    sizeof(expn) - request->stream.avail_out);\n+\t} while (request->stream.avail_in && request->zret == Z_OK);\n \treturn size;\n }\n \n-void prefetch(unsigned char *sha1)\n-{\n-}\n-\n int relink_or_rename(char *old, char *new) {\n \tint ret;\n \n@@ -110,10 +144,296 @@ int relink_or_rename(char *old, char *ne\n \treturn 0;\n }\n \n+struct active_request_slot *get_active_slot()\n+{\n+\tstruct active_request_slot *slot = active_queue_head;\n+\tstruct active_request_slot *newslot;\n+\n+\twhile (slot != NULL && slot->in_use) {\n+\t\tslot = slot->next;\n+\t}\n+\tif (slot == NULL) {\n+\t\tnewslot = xmalloc(sizeof(*newslot));\n+\t\tnewslot->curl = curl_easy_duphandle(curl_default);\n+\t\tnewslot->in_use = 0;\n+\t\tnewslot->next = NULL;\n+\n+\t\tslot = active_queue_head;\n+\t\tif (slot == NULL) {\n+\t\t\tactive_queue_head = newslot;\n+\t\t} else {\n+\t\t\twhile (slot->next != NULL) {\n+\t\t\t\tslot = slot->next;\n+\t\t\t}\n+\t\t\tslot->next = newslot;\n+\t\t}\n+\t\tslot = newslot;\n+\t}\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_range_header);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n+\n+\treturn slot;\n+}\n+\n+void finish_request(struct transfer_request *request)\n+{\n+\tfchmod(request->local, 0444);\n+\tclose(request->local);\n+\n+\tif (request->http_code == 416) {\n+\t\tfprintf(stderr, \"Warning: requested range invalid; we may already have all the data.\\n\");\n+\t} else if (request->curl_result != CURLE_OK) {\n+\t\treturn;\n+\t}\n+\n+\tinflateEnd(&request->stream);\n+\tSHA1_Final(request->real_sha1, &request->c);\n+\tif (request->zret != Z_STREAM_END) {\n+\t\tunlink(request->tmpfile);\n+\t\treturn;\n+\t}\n+\tif (memcmp(request->sha1, request->real_sha1, 20)) {\n+\t\tunlink(request->tmpfile);\n+\t\treturn;\n+\t}\n+\trequest->rename =\n+\t\trelink_or_rename(request->tmpfile, request->filename);\n+}\n+\n+void release_request(struct transfer_request *request)\n+{\n+\tstruct transfer_request *entry = request_queue_head;\n+\n+\tif (request == request_queue_head) {\n+\t\trequest_queue_head = request->next;\n+\t} else {\n+\t\twhile (entry->next != NULL && entry->next != request)\n+\t\t\tentry = entry->next;\n+\t\tif (entry->next == request)\n+\t\t\tentry->next = entry->next->next;\n+\t}\n+\n+\tfree(request->url);\n+\tfree(request);\n+}\n+\n+void start_request(struct transfer_request *request)\n+{\n+\tchar *hex = sha1_to_hex(request->sha1);\n+\tchar prevfile[PATH_MAX];\n+\tchar *url;\n+\tchar *posn;\n+\tint prevlocal;\n+\tunsigned char prev_buf[PREV_BUF_SIZE];\n+\tssize_t prev_read = 0;\n+\tlong prev_posn = 0;\n+\tchar range[RANGE_HEADER_SIZE];\n+\tstruct curl_slist *range_header = NULL;\n+\tCURLMcode curlm_result;\n+\tstruct active_request_slot *slot;\n+\n+\tsnprintf(prevfile, sizeof(prevfile), \"%s.prev\", request->filename);\n+\tunlink(prevfile);\n+\trename(request->tmpfile, prevfile);\n+\tunlink(request->tmpfile);\n+\n+\trequest->local = open(request->tmpfile,\n+\t\t\t      O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\tif (request->local < 0) {\n+\t\trequest->state = ABORTED;\n+\t\terror(\"Couldn't create temporary file %s for %s: %s\\n\",\n+\t\t      request->tmpfile, request->filename, strerror(errno));\n+\t\treturn;\n+\t}\n+\n+\tmemset(&request->stream, 0, sizeof(request->stream));\n+\n+\tinflateInit(&request->stream);\n+\n+\tSHA1_Init(&request->c);\n+\n+\turl = xmalloc(strlen(request->repo->base) + 50);\n+\trequest->url = xmalloc(strlen(request->repo->base) + 50);\n+\tstrcpy(url, request->repo->base);\n+\tposn = url + strlen(request->repo->base);\n+\tstrcpy(posn, \"objects/\");\n+\tposn += 8;\n+\tmemcpy(posn, hex, 2);\n+\tposn += 2;\n+\t*(posn++) = '/';\n+\tstrcpy(posn, hex + 2);\n+\tstrcpy(request->url, url);\n+\n+\tslot = get_active_slot();\n+\tslot->in_use = 1;\n+\trequest->slot = slot;\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, request);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, request->errorstr);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n+\n+\t/* If a previous temp file is present, process what was already\n+\t   fetched. */\n+\tprevlocal = open(prevfile, O_RDONLY);\n+\tif (prevlocal != -1) {\n+\t\tdo {\n+\t\t\tprev_read = read(prevlocal, prev_buf, PREV_BUF_SIZE);\n+\t\t\tif (prev_read>0) {\n+\t\t\t\tif (fwrite_sha1_file(prev_buf,\n+\t\t\t\t\t\t     1,\n+\t\t\t\t\t\t     prev_read,\n+\t\t\t\t\t\t     request) == prev_read) {\n+\t\t\t\t\tprev_posn += prev_read;\n+\t\t\t\t} else {\n+\t\t\t\t\tprev_read = -1;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t} while (prev_read > 0);\n+\t\tclose(prevlocal);\n+\t}\n+\tunlink(prevfile);\n+\n+\t/* Reset inflate/SHA1 if there was an error reading the previous temp\n+\t   file; also rewind to the beginning of the local file. */\n+\tif (prev_read == -1) {\n+\t\tmemset(&request->stream, 0, sizeof(request->stream));\n+\t\tinflateInit(&request->stream);\n+\t\tSHA1_Init(&request->c);\n+\t\tif (prev_posn>0) {\n+\t\t\tprev_posn = 0;\n+\t\t\tlseek(request->local, SEEK_SET, 0);\n+\t\t\tftruncate(request->local, 0);\n+\t\t}\n+\t}\n+\n+\t/* If we have successfully processed data from a previous fetch\n+\t   attempt, only fetch the data we don't already have. */\n+\tif (prev_posn>0) {\n+\t\tif (get_verbosely)\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"Resuming fetch of object %s at byte %ld\\n\",\n+\t\t\t\thex, prev_posn);\n+\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n+\t\trange_header = curl_slist_append(range_header, range);\n+\t\tcurl_easy_setopt(slot->curl,\n+\t\t\t\t CURLOPT_HTTPHEADER, range_header);\n+\t}\n+\n+\t/* Try to add to multi handle, abort the request on error */\n+\tcurlm_result = curl_multi_add_handle(curlm, slot->curl);\n+\tif (curlm_result != CURLM_OK &&\n+\t    curlm_result != CURLM_CALL_MULTI_PERFORM) {\n+\t\trequest->state = ABORTED;\n+\t\tclose(request->local);\n+\t\tfree(request->url);\n+\t\tslot->in_use = 0;\n+\t\treturn;\n+\t}\n+\t\n+\trequest->slot = slot;\n+\trequest->state = ACTIVE;\n+}\n+\n+void process_curl_messages()\n+{\n+\tint num_messages;\n+\tstruct transfer_request *request;\n+\tCURLMsg *curl_message = curl_multi_info_read(curlm, &num_messages);\n+\n+\twhile (curl_message != NULL) {\n+\t\tif (curl_message->msg == CURLMSG_DONE) {\n+\t\t\trequest = request_queue_head;\n+\t\t\twhile (request != NULL) {\n+\t\t\t\tif (request->slot != NULL &&\n+\t\t\t\t    request->slot->curl ==\n+\t\t\t\t    curl_message->easy_handle)\n+\t\t\t\t\tbreak;\n+\t\t\t\trequest = request->next;\n+\t\t\t}\n+\t\t\tif (request != NULL) {\n+\t\t\t\tcurl_multi_remove_handle(curlm,\n+\t\t\t\t\t\t\t request->slot->curl);\n+\t\t\t\trequest->curl_result =\n+\t\t\t\t\tcurl_message->data.result;\n+\t\t\t\tcurl_easy_getinfo(request->slot->curl,\n+\t\t\t\t\t\t  CURLINFO_HTTP_CODE,\n+\t\t\t\t\t\t  &request->http_code);\n+\t\t\t\trequest->slot->in_use = 0;\n+\t\t\t\trequest->slot = NULL;\n+\n+\t\t\t\t/* Use alternates if necessary */\n+\t\t\t\tif (request->http_code == 404 &&\n+\t\t\t\t    request->repo->next != NULL) {\n+\t\t\t\t\trequest->repo = request->repo->next;\n+\t\t\t\t\tstart_request(request);\n+\t\t\t\t} else {\n+\t\t\t\t\tfinish_request(request);\n+\t\t\t\t\trequest->state = COMPLETE;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tfprintf(stderr, \"Received DONE message for unknown request!\\n\");\n+\t\t\t}\n+\t\t} else {\n+\t\t\tfprintf(stderr, \"Unknown CURL message received: %d\\n\",\n+\t\t\t\t(int)curl_message->msg);\n+\t\t}\n+\t\tcurl_message = curl_multi_info_read(curlm, &num_messages);\n+\t}\n+}\n+\n+void process_request_queue()\n+{\n+\tstruct transfer_request *request = request_queue_head;\n+\tint num_transfers;\n+\n+\tcurl_multi_perform(curlm, &num_transfers);\n+\twhile (num_transfers < max_requests && request != NULL) {\n+\t\twhile (request != NULL && request->state != WAITING)\n+\t\t\trequest = request->next;\n+\t\tif (request != NULL) {\n+\t\t\tstart_request(request);\n+\t\t\tcurl_multi_perform(curlm, &num_transfers);\n+\t\t}\n+\t}\n+}\n+\n+void prefetch(unsigned char *sha1)\n+{\n+\tstruct transfer_request *newreq;\n+\tstruct transfer_request *tail;\n+\tchar *filename = sha1_file_name(sha1);\n+\n+\tnewreq = xmalloc(sizeof(*newreq));\n+\tmemcpy(newreq->sha1, sha1, 20);\n+\tnewreq->repo = alt;\n+\tnewreq->url = NULL;\n+\tnewreq->local = -1;\n+\tnewreq->state = WAITING;\n+\tsnprintf(newreq->filename, sizeof(newreq->filename), \"%s\", filename);\n+\tsnprintf(newreq->tmpfile, sizeof(newreq->tmpfile),\n+\t\t \"%s.temp\", filename);\n+\tnewreq->next = NULL;\n+\n+\tif (request_queue_head == NULL) {\n+\t\trequest_queue_head = newreq;\n+\t} else {\n+\t\ttail = request_queue_head;\n+\t\twhile (tail->next != NULL) {\n+\t\t\ttail = tail->next;\n+\t\t}\n+\t\ttail->next = newreq;\n+\t}\n+\tprocess_request_queue();\n+\tprocess_curl_messages();\n+}\n+\n static int got_alternates = 0;\n \n static int fetch_index(struct alt_base *repo, unsigned char *sha1)\n {\n+\tchar *hex = sha1_to_hex(sha1);\n \tchar *filename;\n \tchar *url;\n \tchar tmpfile[PATH_MAX];\n@@ -124,17 +444,16 @@ static int fetch_index(struct alt_base *\n \tCURLcode curl_result;\n \n \tFILE *indexfile;\n+\tstruct active_request_slot *slot = get_active_slot();\n \n \tif (has_pack_index(sha1))\n \t\treturn 0;\n \n \tif (get_verbosely)\n-\t\tfprintf(stderr, \"Getting index for pack %s\\n\",\n-\t\t\tsha1_to_hex(sha1));\n+\t\tfprintf(stderr, \"Getting index for pack %s\\n\", hex);\n \t\n \turl = xmalloc(strlen(repo->base) + 64);\n-\tsprintf(url, \"%s/objects/pack/pack-%s.idx\",\n-\t\trepo->base, sha1_to_hex(sha1));\n+\tsprintf(url, \"%s/objects/pack/pack-%s.idx\", repo->base, hex);\n \t\n \tfilename = sha1_pack_index_name(sha1);\n \tsnprintf(tmpfile, sizeof(tmpfile), \"%s.temp\", filename);\n@@ -143,11 +462,9 @@ static int fetch_index(struct alt_base *\n \t\treturn error(\"Unable to open local file %s for pack index\",\n \t\t\t     filename);\n \n-\tcurl_easy_setopt(curl, CURLOPT_FILE, indexfile);\n-\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n-\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n-\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);\n-\tcurl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, indexfile);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n \t\n \t/* If there is data present from a previous transfer attempt,\n \t   resume where it left off */\n@@ -156,17 +473,14 @@ static int fetch_index(struct alt_base *\n \t\tif (get_verbosely)\n \t\t\tfprintf(stderr,\n \t\t\t\t\"Resuming fetch of index for pack %s at byte %ld\\n\",\n-\t\t\t\tsha1_to_hex(sha1), prev_posn);\n+\t\t\t\thex, prev_posn);\n \t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n \t\trange_header = curl_slist_append(range_header, range);\n-\t\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, range_header);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, range_header);\n \t}\n \n-\t/* Clear out the Range: header after performing the request, so\n-\t   other curl requests don't inherit inappropriate header data */\n-\tcurl_result = curl_easy_perform(curl);\n-\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);\n-\tif (curl_result != 0) {\n+\tcurl_result = curl_easy_perform(slot->curl);\n+\tif (curl_result != CURLE_OK) {\n \t\tfclose(indexfile);\n \t\treturn error(\"Unable to get pack index %s\\n%s\", url,\n \t\t\t     curl_errorstr);\n@@ -205,6 +519,9 @@ static int fetch_alternates(char *base)\n \tchar *data;\n \tint i = 0;\n \tint http_specific = 1;\n+\tstruct alt_base *tail = alt;\n+\n+\tstruct active_request_slot *slot = get_active_slot();\n \tif (got_alternates)\n \t\treturn 0;\n \tdata = xmalloc(4096);\n@@ -218,20 +535,18 @@ static int fetch_alternates(char *base)\n \turl = xmalloc(strlen(base) + 31);\n \tsprintf(url, \"%s/objects/info/http-alternates\", base);\n \n-\tcurl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n-\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n-\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n-\n-\tif (curl_easy_perform(curl) || !buffer.posn) {\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n+\tif (curl_easy_perform(slot->curl) || !buffer.posn) {\n \t\thttp_specific = 0;\n \n \t\tsprintf(url, \"%s/objects/info/alternates\", base);\n \t\t\n-\t\tcurl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n-\t\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n-\t\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n-\t\t\n-\t\tif (curl_easy_perform(curl)) {\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n+\t\tif (curl_easy_perform(slot->curl)) {\n \t\t\treturn 0;\n \t\t}\n \t}\n@@ -283,11 +598,13 @@ static int fetch_alternates(char *base)\n \t\t\t\t\tfprintf(stderr, \n \t\t\t\t\t\t\"Also look at %s\\n\", target);\n \t\t\t\tnewalt = xmalloc(sizeof(*newalt));\n-\t\t\t\tnewalt->next = alt;\n+\t\t\t\tnewalt->next = NULL;\n \t\t\t\tnewalt->base = target;\n \t\t\t\tnewalt->got_indices = 0;\n \t\t\t\tnewalt->packs = NULL;\n-\t\t\t\talt = newalt;\n+\t\t\t\twhile (tail->next != NULL)\n+\t\t\t\t\ttail = tail->next;\n+\t\t\t\ttail->next = newalt;\n \t\t\t\tret++;\n \t\t\t}\n \t\t}\n@@ -306,6 +623,8 @@ static int fetch_indices(struct alt_base\n \tchar *data;\n \tint i = 0;\n \n+\tstruct active_request_slot *slot = get_active_slot();\n+\n \tif (repo->got_indices)\n \t\treturn 0;\n \n@@ -320,13 +639,12 @@ static int fetch_indices(struct alt_base\n \turl = xmalloc(strlen(repo->base) + 21);\n \tsprintf(url, \"%s/objects/info/packs\", repo->base);\n \n-\tcurl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n-\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n-\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n-\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, NULL);\n-\tcurl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, NULL);\n \t\n-\tif (curl_easy_perform(curl))\n+\tif (curl_easy_perform(slot->curl))\n \t\treturn error(\"%s\", curl_errorstr);\n \n \twhile (i < buffer.posn) {\n@@ -366,6 +684,8 @@ static int fetch_pack(struct alt_base *r\n \tstruct curl_slist *range_header = NULL;\n \tCURLcode curl_result;\n \n+\tstruct active_request_slot *slot = get_active_slot();\n+\n \tif (fetch_indices(repo))\n \t\treturn -1;\n \ttarget = find_sha1_pack(sha1, repo->packs);\n@@ -390,11 +710,9 @@ static int fetch_pack(struct alt_base *r\n \t\treturn error(\"Unable to open local file %s for pack\",\n \t\t\t     filename);\n \n-\tcurl_easy_setopt(curl, CURLOPT_FILE, packfile);\n-\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n-\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n-\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);\n-\tcurl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, packfile);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n \n \t/* If there is data present from a previous transfer attempt,\n \t   resume where it left off */\n@@ -406,14 +724,11 @@ static int fetch_pack(struct alt_base *r\n \t\t\t\tsha1_to_hex(target->sha1), prev_posn);\n \t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n \t\trange_header = curl_slist_append(range_header, range);\n-\t\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, range_header);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, range_header);\n \t}\n \n-\t/* Clear out the Range: header after performing the request, so\n-\t   other curl requests don't inherit inappropriate header data */\n-\tcurl_result = curl_easy_perform(curl);\n-\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);\n-\tif (curl_result != 0) {\n+\tcurl_result = curl_easy_perform(slot->curl);\n+\tif (curl_result != CURLE_OK) {\n \t\tfclose(packfile);\n \t\treturn error(\"Unable to get pack file %s\\n%s\", url,\n \t\t\t     curl_errorstr);\n@@ -441,135 +756,92 @@ static int fetch_pack(struct alt_base *r\n static int fetch_object(struct alt_base *repo, unsigned char *sha1)\n {\n \tchar *hex = sha1_to_hex(sha1);\n-\tchar *filename = sha1_file_name(sha1);\n-\tunsigned char real_sha1[20];\n-\tchar tmpfile[PATH_MAX];\n-\tchar prevfile[PATH_MAX];\n \tint ret;\n-\tchar *url;\n-\tchar *posn;\n-\tint prevlocal;\n-\tunsigned char prev_buf[PREV_BUF_SIZE];\n-\tssize_t prev_read = 0;\n-\tlong prev_posn = 0;\n-\tchar range[RANGE_HEADER_SIZE];\n-\tstruct curl_slist *range_header = NULL;\n-\tCURLcode curl_result;\n-\n-\tsnprintf(tmpfile, sizeof(tmpfile), \"%s.temp\", filename);\n-\tsnprintf(prevfile, sizeof(prevfile), \"%s.prev\", filename);\n-\n-\tif (unlink(prevfile) && (errno != ENOENT))\n-\t\treturn error(\"Failed to unlink %s (%s)\",\n-\t\t\t     prevfile, strerror(errno));\n-\tif (rename(tmpfile, prevfile) && (errno != ENOENT))\n-\t\treturn error(\"Failed to rename %s to %s (%s)\",\n-\t\t\t     tmpfile, prevfile, strerror(errno));\n-\n-\tlocal = open(tmpfile, O_WRONLY | O_CREAT | O_EXCL, 0666);\n-\n-\t/* Note: if another instance starts now, it will turn our new\n-\t   tmpfile into its prevfile. */\n-\n-\tif (local < 0)\n-\t\treturn error(\"Couldn't create temporary file %s for %s: %s\\n\",\n-\t\t\t     tmpfile, filename, strerror(errno));\n-\n-\tmemset(&stream, 0, sizeof(stream));\n-\n-\tinflateInit(&stream);\n-\n-\tSHA1_Init(&c);\n+\tstruct transfer_request *request = request_queue_head;\n+\tint num_transfers;\n+\tint num_remaining;\n+\tfd_set readfds;\n+\tfd_set writefds;\n+\tfd_set excfds;\n+\tint max_fd;\n+\tstruct timeval select_timeout;\n+\tdouble download_size;\n+\tdouble current_size;\n \n-\tcurl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);\n-\tcurl_easy_setopt(curl, CURLOPT_FILE, NULL);\n-\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n-\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);\n-\tcurl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n-\n-\turl = xmalloc(strlen(repo->base) + 50);\n-\tstrcpy(url, repo->base);\n-\tposn = url + strlen(repo->base);\n-\tstrcpy(posn, \"objects/\");\n-\tposn += 8;\n-\tmemcpy(posn, hex, 2);\n-\tposn += 2;\n-\t*(posn++) = '/';\n-\tstrcpy(posn, hex + 2);\n-\n-\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\twhile (request != NULL && memcmp(request->sha1, sha1, 20)) {\n+\t\trequest = request->next;\n+\t}\n+\tif (request == NULL)\n+\t\treturn error(\"Couldn't find request for %s in the queue\", hex);\n \n-\t/* If a previous temp file is present, process what was already\n-\t   fetched. */\n-\tprevlocal = open(prevfile, O_RDONLY);\n-\tif (prevlocal != -1) {\n+\tdo {\n+\t\tdo {} while (curl_multi_perform(curlm, &num_transfers) ==\n+\t\t\t     CURLM_CALL_MULTI_PERFORM);\n+\t\tprocess_curl_messages();\n+\t\tprocess_request_queue();\n+\t} while (request->state == WAITING);\n+\n+\twhile (request->state == ACTIVE) {\n+\t\tcurl_easy_getinfo(request->slot->curl,\n+\t\t\t\t  CURLINFO_SIZE_DOWNLOAD,\n+\t\t\t\t  &download_size);\n \t\tdo {\n-\t\t\tprev_read = read(prevlocal, prev_buf, PREV_BUF_SIZE);\n-\t\t\tif (prev_read>0) {\n-\t\t\t\tif (fwrite_sha1_file(prev_buf,\n-\t\t\t\t\t\t     1,\n-\t\t\t\t\t\t     prev_read,\n-\t\t\t\t\t\t     NULL) == prev_read) {\n-\t\t\t\t\tprev_posn += prev_read;\n-\t\t\t\t} else {\n-\t\t\t\t\tprev_read = -1;\n-\t\t\t\t}\n+\t\t\tnum_remaining = num_transfers;\n+\t\t\tcurrent_size = download_size;\n+\t\t\tcurl_multi_perform(curlm, &num_transfers);\n+\t\t\tcurl_easy_getinfo(request->slot->curl,\n+\t\t\t\t\t  CURLINFO_SIZE_DOWNLOAD,\n+\t\t\t\t\t  &download_size);\n+\t\t\tif (num_remaining != num_transfers) {\n+\t\t\t\tprocess_curl_messages();\n+\t\t\t\tprocess_request_queue();\n \t\t\t}\n-\t\t} while (prev_read > 0);\n-\t\tclose(prevlocal);\n-\t}\n-\tunlink(prevfile);\n+\t\t} while (download_size > current_size &&\n+\t\t\t request->state == ACTIVE);\n \n-\t/* Reset inflate/SHA1 if there was an error reading the previous temp\n-\t   file; also rewind to the beginning of the local file. */\n-\tif (prev_read == -1) {\n-\t\tmemset(&stream, 0, sizeof(stream));\n-\t\tinflateInit(&stream);\n-\t\tSHA1_Init(&c);\n-\t\tif (prev_posn>0) {\n-\t\t\tprev_posn = 0;\n-\t\t\tlseek(local, SEEK_SET, 0);\n-\t\t\tftruncate(local, 0);\n+\t\tif (request->state == ACTIVE) {\n+\t\t\tFD_ZERO(&readfds);\n+\t\t\tFD_ZERO(&writefds);\n+\t\t\tFD_ZERO(&excfds);\n+\t\t\tmax_fd = 0;\n+\t\t\tselect_timeout.tv_sec = 0;\n+\t\t\tselect_timeout.tv_usec = 50000;\n+\t\t\tselect(max_fd, &readfds, &writefds,\n+\t\t\t       &excfds, &select_timeout);\n \t\t}\n \t}\n \n-\t/* If we have successfully processed data from a previous fetch\n-\t   attempt, only fetch the data we don't already have. */\n-\tif (prev_posn>0) {\n-\t\tif (get_verbosely)\n-\t\t\tfprintf(stderr,\n-\t\t\t\t\"Resuming fetch of object %s at byte %ld\\n\",\n-\t\t\t\thex, prev_posn);\n-\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n-\t\trange_header = curl_slist_append(range_header, range);\n-\t\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, range_header);\n+\tif (request->state == ABORTED) {\n+\t\trelease_request(request);\n+\t\treturn error(\"Request for %s aborted\", hex);\n \t}\n \n-\t/* Clear out the Range: header after performing the request, so\n-\t   other curl requests don't inherit inappropriate header data */\n-\tcurl_result = curl_easy_perform(curl);\n-\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);\n-\tif (curl_result != 0) {\n-\t\treturn error(\"%s\", curl_errorstr);\n+\tif (request->curl_result != CURLE_OK && request->http_code != 416) {\n+\t\tret = error(\"%s\", request->errorstr);\n+\t\trelease_request(request);\n+\t\treturn ret;\n \t}\n \n-\tfchmod(local, 0444);\n-\tclose(local);\n-\tinflateEnd(&stream);\n-\tSHA1_Final(real_sha1, &c);\n-\tif (zret != Z_STREAM_END) {\n-\t\tunlink(tmpfile);\n-\t\treturn error(\"File %s (%s) corrupt\\n\", hex, url);\n+\tif (request->zret != Z_STREAM_END) {\n+\t\tret = error(\"File %s (%s) corrupt\\n\", hex, request->url);\n+\t\trelease_request(request);\n+\t\treturn ret;\n \t}\n-\tif (memcmp(sha1, real_sha1, 20)) {\n-\t\tunlink(tmpfile);\n+\n+\tif (memcmp(request->sha1, request->real_sha1, 20)) {\n+\t\trelease_request(request);\n \t\treturn error(\"File %s has bad hash\\n\", hex);\n \t}\n-\tret = relink_or_rename(tmpfile, filename);\n-\tif (ret)\n-\t\treturn error(\"unable to write sha1 filename %s: %s\",\n-\t\t\t     filename, strerror(ret));\n \n+\tif (request->rename < 0) {\n+\t\tret = error(\"unable to write sha1 filename %s: %s\",\n+\t\t\t    request->filename,\n+\t\t\t    strerror(request->rename));\n+\t\trelease_request(request);\n+\t\treturn ret;\n+\t}\n+\n+\trelease_request(request);\n \tpull_say(\"got %s\\n\", hex);\n \treturn 0;\n }\n@@ -577,19 +849,16 @@ static int fetch_object(struct alt_base \n int fetch(unsigned char *sha1)\n {\n \tstruct alt_base *altbase = alt;\n+\n+\tif (!fetch_object(altbase, sha1))\n+\t\treturn 0;\n \twhile (altbase) {\n-\t\tif (!fetch_object(altbase, sha1))\n-\t\t\treturn 0;\n \t\tif (!fetch_pack(altbase, sha1))\n \t\t\treturn 0;\n-\t\tif (fetch_alternates(altbase->base) > 0) {\n-\t\t\taltbase = alt;\n-\t\t\tcontinue;\n-\t\t}\n \t\taltbase = altbase->next;\n \t}\n \treturn error(\"Unable to find %s under %s\\n\", sha1_to_hex(sha1), \n-\t\t     initial_base);\n+\t\t     alt->base);\n }\n \n int fetch_ref(char *ref, unsigned char *sha1)\n@@ -597,16 +866,16 @@ int fetch_ref(char *ref, unsigned char *\n         char *url, *posn;\n         char hex[42];\n         struct buffer buffer;\n-\tchar *base = initial_base;\n+\tchar *base = alt->base;\n+\tstruct active_request_slot *slot = get_active_slot();\n         buffer.size = 41;\n         buffer.posn = 0;\n         buffer.buffer = hex;\n         hex[41] = '\\0';\n         \n-        curl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n-        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n-\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, NULL);\n-\tcurl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);\n+        curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, NULL);\n \n         url = xmalloc(strlen(base) + 6 + strlen(ref));\n         strcpy(url, base);\n@@ -615,9 +884,9 @@ int fetch_ref(char *ref, unsigned char *\n         posn += 5;\n         strcpy(posn, ref);\n \n-        curl_easy_setopt(curl, CURLOPT_URL, url);\n+        curl_easy_setopt(slot->curl, CURLOPT_URL, url);\n \n-        if (curl_easy_perform(curl))\n+        if (curl_easy_perform(slot->curl))\n                 return error(\"Couldn't get %s for %s\\n%s\",\n \t\t\t     url, ref, curl_errorstr);\n \n@@ -631,6 +900,7 @@ int main(int argc, char **argv)\n \tchar *commit_id;\n \tchar *url;\n \tint arg = 1;\n+\tstruct active_request_slot *slot;\n \n \twhile (arg < argc && argv[arg][0] == '-') {\n \t\tif (argv[arg][1] == 't') {\n@@ -648,6 +918,11 @@ int main(int argc, char **argv)\n \t\t\targ++;\n \t\t} else if (!strcmp(argv[arg], \"--recover\")) {\n \t\t\tget_recover = 1;\n+\t\t} else if (argv[arg][1] == 'r') {\n+\t\t\tmax_requests = atoi(argv[arg + 1]);\n+\t\t\tif (max_requests < 1)\n+\t\t\t\tmax_requests = DEFAULT_MAX_REQUESTS;\n+\t\t\targ++;\n \t\t}\n \t\targ++;\n \t}\n@@ -660,44 +935,61 @@ int main(int argc, char **argv)\n \n \tcurl_global_init(CURL_GLOBAL_ALL);\n \n-\tcurl = curl_easy_init();\n+\tcurlm = curl_multi_init();\n+\tif (curlm == NULL)\n+\t\tfprintf(stderr, \"Error creating curl multi handle.\\n\");\n \tno_pragma_header = curl_slist_append(no_pragma_header, \"Pragma:\");\n \tno_range_header = curl_slist_append(no_range_header, \"Range:\");\n \n \tcurl_ssl_verify = getenv(\"GIT_SSL_NO_VERIFY\") ? 0 : 1;\n-\tcurl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, curl_ssl_verify);\n+\tssl_cert = getenv(\"GIT_SSL_CERT\");\n+\tssl_key = getenv(\"GIT_SSL_KEY\");\n+\tssl_capath = getenv(\"GIT_SSL_CAPATH\");\n+\tssl_cainfo = getenv(\"GIT_SSL_CAINFO\");\n+\n+\tcurl_default = curl_easy_init();\n+\tcurl_easy_setopt(curl_default, CURLOPT_SSL_VERIFYPEER, curl_ssl_verify);\n #if LIBCURL_VERSION_NUM >= 0x070907\n-\tcurl_easy_setopt(curl, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);\n+\tcurl_easy_setopt(curl_default, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);\n #endif\n \n-\tif ((ssl_cert = getenv(\"GIT_SSL_CERT\")) != NULL) {\n-\t\tcurl_easy_setopt(curl, CURLOPT_SSLCERT, ssl_cert);\n+\tif (ssl_cert != NULL) {\n+\t\tcurl_easy_setopt(curl_default, CURLOPT_SSLCERT, ssl_cert);\n \t}\n #if LIBCURL_VERSION_NUM >= 0x070902\n-\tif ((ssl_key = getenv(\"GIT_SSL_KEY\")) != NULL) {\n-\t\tcurl_easy_setopt(curl, CURLOPT_SSLKEY, ssl_key);\n+\tif (ssl_key != NULL) {\n+\t\tcurl_easy_setopt(curl_default, CURLOPT_SSLKEY, ssl_key);\n \t}\n #endif\n #if LIBCURL_VERSION_NUM >= 0x070908\n-\tif ((ssl_capath = getenv(\"GIT_SSL_CAPATH\")) != NULL) {\n-\t\tcurl_easy_setopt(curl, CURLOPT_CAPATH, ssl_capath);\n+\tif (ssl_capath != NULL) {\n+\t\tcurl_easy_setopt(curl_default, CURLOPT_CAPATH, ssl_capath);\n \t}\n #endif\n-\tif ((ssl_cainfo = getenv(\"GIT_SSL_CAINFO\")) != NULL) {\n-\t\tcurl_easy_setopt(curl, CURLOPT_CAINFO, ssl_cainfo);\n+\tif (ssl_cainfo != NULL) {\n+\t\tcurl_easy_setopt(curl_default, CURLOPT_CAINFO, ssl_cainfo);\n \t}\n+\tcurl_easy_setopt(curl_default, CURLOPT_FAILONERROR, 1);\n \n \talt = xmalloc(sizeof(*alt));\n \talt->base = url;\n \talt->got_indices = 0;\n \talt->packs = NULL;\n \talt->next = NULL;\n-\tinitial_base = url;\n+\tfetch_alternates(alt->base);\n \n \tif (pull(commit_id))\n \t\treturn 1;\n \n \tcurl_slist_free_all(no_pragma_header);\n+\tcurl_slist_free_all(no_range_header);\n+\tcurl_easy_cleanup(curl_default);\n+\tslot = active_queue_head;\n+\twhile (slot != NULL) {\n+\t\tcurl_easy_cleanup(slot->curl);\n+\t\tslot = slot->next;\n+\t}\n+\tcurl_multi_cleanup(curlm);\n \tcurl_global_cleanup();\n \treturn 0;\n }\n"},{"id":"9773","messageId":"Pine.LNX.4.63.0510061550510.23242@iabervon.org","threadId":"2025","inReplyTo":"20051005214447.GF15593@reactrix.com","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-10-06T20:07:07Z","receivedAt":"2005-10-06T20:07:07Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Wed, 5 Oct 2005, Nick Hengeveld wrote:\n\n> Add support for parallel HTTP transfers.  Prefetch populates a queue of\n> objects to transfer and starts feeding requests to an active request\n> queue for processing; fetch_object keeps the active queue moving\n> while the specified object is being transferred.  The size of the active\n> queue can be restricted using -r and defaults to 5 concurrent transfers.\n\nSomewhat weirdly, the version of curl on my desktop doesn't actually have \nan implementation of curl_multi_info_read, although it's in the header \nfile and documentation. So you'll want a version check somewhere, I think, \nwhich should probably just disable parallel transfers.\n\n> ---\n> \n> I could use extra eyes on this patch - it seems to be stable although I have\n> seen periodic cases during testing where it detects an empty server response\n> (from kernel.org?)  Are there implications to downloading a (potentially large)\n> pack while objects contained in that pack have been prefetched and are in the\n> transfer and/or active queue?\n\nIt should be fine to download objects and a pack that contains them at the \nsame time, although there's currently a check in fetch.c which should be \nremoved, so that it will call fetch() for an object if the object appears \nbetween the prefetch() and the fetch().\n\nI should be able to review this over the weekend. What sort of performance \nare you getting at this point (in terms of bandwidth utilization)?\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"9784","messageId":"20051007000041.GH15593@reactrix.com","threadId":"2025","inReplyTo":"Pine.LNX.4.63.0510061550510.23242@iabervon.org","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-10-07T00:00:41Z","receivedAt":"2005-10-07T00:00:41Z","isPatch":true,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"On Thu, Oct 06, 2005 at 04:07:07PM -0400, Daniel Barkalow wrote:\n\n> Somewhat weirdly, the version of curl on my desktop doesn't actually have \n> an implementation of curl_multi_info_read, although it's in the header \n> file and documentation. So you'll want a version check somewhere, I think, \n> which should probably just disable parallel transfers.\n\nI was afraid that was going to happen...  From the archived versions on the\nCURL download site, it looks as though multi support was added in 7.9.8 -\nwhich version do you have installed on your desktop?\n\nI'll follow this up with a patch that works as you describe to disable\nbuilding with parallel transfer support on versions < 7.9.8.\n\n> It should be fine to download objects and a pack that contains them at the \n> same time, although there's currently a check in fetch.c which should be \n> removed, so that it will call fetch() for an object if the object appears \n> between the prefetch() and the fetch().\n\nCan you provide a patch, or point me toward the right place to make that\nchange?\n\n> I should be able to review this over the weekend. What sort of performance \n> are you getting at this point (in terms of bandwidth utilization)?\n\nI've done limited testing by using the time command to track real/user/sys\ntaken to run 'git fetch http://kernel.org/pub/scm/git/git.git master',\nand have seen performance improve by a factor of ~2-10:\n\n0.99.8:\n     real 2m48.800s,  user 0m2.540s, sys 0m0.470s\n     real 2m40.316s,  user 0m2.850s, sys 0m0.500s\n     real 2m8.543s,   user 0m2.910s, sys 0m0.600s\n     real 2m18.009s,  user 0m2.440s, sys 0m0.580s\n     real 1m55.354s,  user 0m2.520s, sys 0m0.430s\n\nParallel: -r 5 (default)\n     real 0m49.499s,  user 0m3.220s, sys 0m0.370s\n     real 1m0.177s,   user 0m3.310s, sys 0m0.740s\n     real 0m52.936s,  user 0m2.680s, sys 0m0.230s\n     real 1m0.158s,   user 0m2.870s, sys 0m0.770s\n     real 0m52.780s,  user 0m2.970s, sys 0m0.600s\n\nParallel: -r 10\n     real 0m28.338s,  user 0m2.940s, sys 0m0.630s\n     real 0m35.944s,  user 0m3.030s, sys 0m0.570s\n     real 0m18.019s,  user 0m3.050s, sys 0m0.530s\n     real 0m21.539s,  user 0m2.960s, sys 0m0.520s\n     real 0m31.405s,  user 0m3.080s, sys 0m0.610s\n\nParallel: -r 20\n     real 0m25.810s,  user 0m3.070s, sys 0m0.490s\n     real 0m16.265s,  user 0m2.880s, sys 0m0.370s\n     real 0m28.536s,  user 0m2.890s, sys 0m0.650s\n     real 0m16.889s,  user 0m2.770s, sys 0m0.460s\n     real 0m23.125s,  user 0m2.800s, sys 0m0.450s\n\nParallel: multi disabled\n     real 3m3.833s,  user 0m12.080s, sys 0m3.240s\n     real 2m15.454s, user 0m12.130s, sys 0m2.820s\n     real 2m23.011s, user 0m12.690s, sys 0m3.030s\n     real 2m38.720s, user 0m12.300s, sys 0m2.850s\n     real 2m42.025s, usre 0m12.310s, sys 0m2.880s\n\nThat's all running on a CentOS 3.5 desktop with CURL 7.10.6.\n\nAbout that \"-r\" arg - seems like it should be something else as -r is\nused elsewhere in git to enable recursion.  \"-c\" was my first thought,\nbut that's used to fetch commit objects.\n\n-- \nFor a successful technology, reality must take precedence over public\nrelations, for nature cannot be fooled.\n"},{"id":"9786","messageId":"7virwa5ety.fsf@assigned-by-dhcp.cox.net","threadId":"2025","inReplyTo":"20051007000041.GH15593@reactrix.com","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-07T00:51:53Z","receivedAt":"2005-10-07T00:51:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nick Hengeveld <nickh@reactrix.com> writes:\n\n> About that \"-r\" arg - seems like it should be something else as -r is\n> used elsewhere in git to enable recursion.  \"-c\" was my first thought,\n> but that's used to fetch commit objects.\n\nWell, I'd suggest just to hardcode a reasonable value to be a\ngood net citizen, and not make it configurable.  Four, perhaps?\n\nOTOH, we may want to have an option to disable parallel from the\ncommand line (I think -r 1 would mean that with yours).\n\nIf we really want to have the number of parallel configurable,\nand -r implies recursive as you say, maybe '-j' to mimic\nparallel make?\n"},{"id":"9789","messageId":"20051007045639.GA18998@reactrix.com","threadId":"2025","inReplyTo":"7virwa5ety.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-10-07T04:56:39Z","receivedAt":"2005-10-07T04:56:39Z","isPatch":true,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"On Thu, Oct 06, 2005 at 05:51:53PM -0700, Junio C Hamano wrote:\n\n> Well, I'd suggest just to hardcode a reasonable value to be a\n> good net citizen, and not make it configurable.  Four, perhaps?\n> \n> OTOH, we may want to have an option to disable parallel from the\n> command line (I think -r 1 would mean that with yours).\n\nI'd prefer to keep it configurable - for our purposes we'll be hitting\na single server from several clients and will probably want to limit\nconcurrent connections to something like two per client, but when doing\na fetch from a big server farm more connections would make sense.\n\n> If we really want to have the number of parallel configurable,\n> and -r implies recursive as you say, maybe '-j' to mimic\n> parallel make?\n\nNot that I'm a huge fan of using environment variables, but it might make\nsense to use one here.  That would allow the setting to work whether\ngit-http-fetch is run directly or via git-fetch.  GIT_HTTP_MAX_REQUESTS?\n\n-- \nFor a successful technology, reality must take precedence over public\nrelations, for nature cannot be fooled.\n"},{"id":"9790","messageId":"7vwtkp29hd.fsf@assigned-by-dhcp.cox.net","threadId":"2025","inReplyTo":"20051007045639.GA18998@reactrix.com","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-07T05:15:42Z","receivedAt":"2005-10-07T05:15:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nick Hengeveld <nickh@reactrix.com> writes:\n\n> Not that I'm a huge fan of using environment variables, but it might make\n> sense to use one here.  That would allow the setting to work whether\n> git-http-fetch is run directly or via git-fetch.  GIT_HTTP_MAX_REQUESTS?\n\nGood point.\n"},{"id":"9798","messageId":"Pine.LNX.4.63.0510071149550.23242@iabervon.org","threadId":"2025","inReplyTo":"20051007000041.GH15593@reactrix.com","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-10-07T16:23:01Z","receivedAt":"2005-10-07T16:23:01Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Thu, 6 Oct 2005, Nick Hengeveld wrote:\n\n> On Thu, Oct 06, 2005 at 04:07:07PM -0400, Daniel Barkalow wrote:\n> \n> > Somewhat weirdly, the version of curl on my desktop doesn't actually have \n> > an implementation of curl_multi_info_read, although it's in the header \n> > file and documentation. So you'll want a version check somewhere, I think, \n> > which should probably just disable parallel transfers.\n> \n> I was afraid that was going to happen...  From the archived versions on the\n> CURL download site, it looks as though multi support was added in 7.9.8 -\n> which version do you have installed on your desktop?\n\nI'll have to check, but I think it's 7.9.8 or close to that; it seems like \nthey added multi support without a critical function, so you might need to \nbump the check from what the history would suggest.\n\n> > It should be fine to download objects and a pack that contains them at the \n> > same time, although there's currently a check in fetch.c which should be \n> > removed, so that it will call fetch() for an object if the object appears \n> > between the prefetch() and the fetch().\n> \n> Can you provide a patch, or point me toward the right place to make that\n> change?\n\nIt's line 168 of fetch.c; the \"!has_sha1_file(obj->sha1)\" part should go \naway.\n\n> > I should be able to review this over the weekend. What sort of performance \n> > are you getting at this point (in terms of bandwidth utilization)?\n> \n> I've done limited testing by using the time command to track real/user/sys\n> taken to run 'git fetch http://kernel.org/pub/scm/git/git.git master',\n> and have seen performance improve by a factor of ~2-10:\n\nThat looks good. I think it might be good to set the default connection \nlimit higher; I don't think we can generate enough parallelism that we'd \ncause problems for a server with a single client, and, with a constant \nstream of clients, this will just shuffle around when the connections \nhappen; to the extent that a single client does more simultaneous \nconnections, it'll overlap less with other clients.\n\n> About that \"-r\" arg - seems like it should be something else as -r is\n> used elsewhere in git to enable recursion.  \"-c\" was my first thought,\n> but that's used to fetch commit objects.\n\nThe other things that affect the behaviour of the HTTP fetch in particular \nare done as environment variables, which seems like a good idea to me. \nAlternatively, you could use a long option. I don't expect there will be \nmuch variation in what someone uses.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"9800","messageId":"7v8xx55kia.fsf@assigned-by-dhcp.cox.net","threadId":"2025","inReplyTo":"Pine.LNX.4.63.0510071149550.23242@iabervon.org","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-07T17:01:33Z","receivedAt":"2005-10-07T17:01:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> writes:\n\n>> > It should be fine to download objects and a pack that contains them at the \n>> > same time, although there's currently a check in fetch.c which should be \n>> > removed, so that it will call fetch() for an object if the object appears \n>> > between the prefetch() and the fetch().\n>> \n>> Can you provide a patch, or point me toward the right place to make that\n>> change?\n>\n> It's line 168 of fetch.c; the \"!has_sha1_file(obj->sha1)\" part should go \n> away.\n\nThe check was added in 029f6de377c7e0484f5c4cf070934599580f1784\nbecause back then calling fetch() on an object that we already\nhad had a funny interaction with what http-fetch.c did.  I\nsuspect that Nick's curl-multi changes made it unnecessary, but\nyou should double check for other transports.\n"},{"id":"9801","messageId":"20051007172206.GK15593@reactrix.com","threadId":"2025","inReplyTo":"7v8xx55kia.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Nick Hengeveld","fromEmail":"nickh@reactrix.com","sentAt":"2005-10-07T17:22:06Z","receivedAt":"2005-10-07T17:22:06Z","isPatch":true,"sender":{"key":"nickh@reactrix.com","avatar":null},"body":"On Fri, Oct 07, 2005 at 10:01:33AM -0700, Junio C Hamano wrote:\n\n> The check was added in 029f6de377c7e0484f5c4cf070934599580f1784\n> because back then calling fetch() on an object that we already\n> had had a funny interaction with what http-fetch.c did.  I\n> suspect that Nick's curl-multi changes made it unnecessary, but\n> you should double check for other transports.\n\nI think the only downside to leaving that check in place is that when\npull() finishes there may be completed requests left behind in the\nqueue, possibly with unreported transfer errors.  Would it make sense\nto just release any requests left in the queue after pull(), and report\nif any of them had transfer errors?\n\n-- \nFor a successful technology, reality must take precedence over public\nrelations, for nature cannot be fooled.\n"},{"id":"9802","messageId":"Pine.LNX.4.63.0510071323070.23242@iabervon.org","threadId":"2025","inReplyTo":"7v8xx55kia.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-10-07T17:41:48Z","receivedAt":"2005-10-07T17:41:48Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Fri, 7 Oct 2005, Junio C Hamano wrote:\n\n> Daniel Barkalow <barkalow@iabervon.org> writes:\n> \n> >> > It should be fine to download objects and a pack that contains them at the \n> >> > same time, although there's currently a check in fetch.c which should be \n> >> > removed, so that it will call fetch() for an object if the object appears \n> >> > between the prefetch() and the fetch().\n> >> \n> >> Can you provide a patch, or point me toward the right place to make that\n> >> change?\n> >\n> > It's line 168 of fetch.c; the \"!has_sha1_file(obj->sha1)\" part should go \n> > away.\n> \n> The check was added in 029f6de377c7e0484f5c4cf070934599580f1784\n> because back then calling fetch() on an object that we already\n> had had a funny interaction with what http-fetch.c did.  I\n> suspect that Nick's curl-multi changes made it unnecessary, but\n> you should double check for other transports.\n\nHmm; my intended convention was that fetch() would always be called if \nprefetch() was called, even if something had happened to make it appear in \nbetween (e.g., prefetch() causing it to be fetched or a different call to \nfetch() speculatively also getting it).\n\nThe ssh transport actually wants to not have the check (if the object \nappears out of nowhere after we request it, we still want to read it out \nof the connection). \n\nThe local transport probably ought to have the check added on line 169 of \nlocal-fetch.c\n\nIn general, transports need to deal with this case themselves, because the \ncore code doesn't know if they started something in prefetch() than needs \nto get finished in fetch().\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"9803","messageId":"7vhdbt42um.fsf@assigned-by-dhcp.cox.net","threadId":"2025","inReplyTo":"Pine.LNX.4.63.0510071323070.23242@iabervon.org","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-07T18:08:17Z","receivedAt":"2005-10-07T18:08:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> writes:\n\n>> The check was added in 029f6de377c7e0484f5c4cf070934599580f1784\n>> because back then calling fetch() on an object that we already\n>> had had a funny interaction with what http-fetch.c did.  I\n>> suspect that Nick's curl-multi changes made it unnecessary, but\n>> you should double check for other transports.\n>\n> Hmm; my intended convention was that fetch() would always be called if \n> prefetch() was called, even if something had happened to make it appear in \n> between (e.g., prefetch() causing it to be fetched or a different call to \n> fetch() speculatively also getting it).\n\nWhen I re-read the code, I think that check was probably a wrong\nfix to begin with.\n\nThe original problem sequence, when the http-fetch was still\nsynchronous, was this:\n\n (1) we ask for an object, fetch_object() did not find one and\n     fetch_pack() got a pack that contained the object and\n     installed it;  the pack is removed from the \"yet to be\n     downloaded from this repository\" list.\n\n (2) we ask for another object, fetch_object() did not find one\n     and fetch_pack() was asked to see if there is a pack we\n     have not downloaded that contained the object -- the pack\n     downloaded in step (1) did not count, and this request\n     failed.  Overall fetch() said \"Nope, I cannot get it\", when\n     it already had one.\n\nWe should remove that check as you suggested, and fix fetch()\nimplementation in http-fetch.c to notice the above situation,\nperhaps?\n"},{"id":"9804","messageId":"7vachl42tt.fsf@assigned-by-dhcp.cox.net","threadId":"2025","inReplyTo":"20051007172206.GK15593@reactrix.com","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-07T18:08:46Z","receivedAt":"2005-10-07T18:08:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nick Hengeveld <nickh@reactrix.com> writes:\n\n> I think the only downside to leaving that check in place is that when\n> pull() finishes there may be completed requests left behind in the\n> queue, possibly with unreported transfer errors.  Would it make sense\n> to just release any requests left in the queue after pull(), and report\n> if any of them had transfer errors?\n\nPull finishing and reporting success while some requests have\nstill been outstanding with transfer errors sounds to me that\ndecision to finish and declare success is made prematurely.\nWhat do these leftover requests you are worried about ask for?\nAre you making redundant requests, which can turn out to be\nunneeded?\n"},{"id":"9815","messageId":"Pine.LNX.4.63.0510071808071.23242@iabervon.org","threadId":"2025","inReplyTo":"7vachl42tt.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-10-07T22:39:22Z","receivedAt":"2005-10-07T22:39:22Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Fri, 7 Oct 2005, Junio C Hamano wrote:\n\n> Nick Hengeveld <nickh@reactrix.com> writes:\n> \n> > I think the only downside to leaving that check in place is that when\n> > pull() finishes there may be completed requests left behind in the\n> > queue, possibly with unreported transfer errors.  Would it make sense\n> > to just release any requests left in the queue after pull(), and report\n> > if any of them had transfer errors?\n> \n> Pull finishing and reporting success while some requests have\n> still been outstanding with transfer errors sounds to me that\n> decision to finish and declare success is made prematurely.\n> What do these leftover requests you are worried about ask for?\n> Are you making redundant requests, which can turn out to be\n> unneeded?\n\nI believe that the situation is the one you describe in your previous \nmessage: we determine we need to fetch A and B; we ask for A; we ask for \nB; we find A isn't available alone, but is available in a pack; we get the \npack; we find we now have B (in the pack); the request for B (which would \nprobably fail) is left dangling.\n\nThe only actual problem I can see is if this happens with a whole bunch of \nobjects at the beginning of a big download, and all but one of your \nconnections are left in this state while you download all of the loose \nobjects over the one connection that got the pack.\n\nI don't know if this is a problem for the new http code, but it could be \nan issue in general if a transport method allocates resources in \nprefetch().\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"9890","messageId":"1128962922.9069.11.camel@cashmere.sps.mot.com","threadId":"2025","inReplyTo":"Pine.LNX.4.63.0510071808071.23242@iabervon.org","subject":"Re: [PATCH] Add support for parallel HTTP transfers","fromName":"Jon Loeliger","fromEmail":"jdl@freescale.com","sentAt":"2005-10-10T16:48:42Z","receivedAt":"2005-10-10T16:48:42Z","isPatch":true,"sender":{"key":"jdl@jdl.com","avatar":"https://gravatar.com/avatar/75ce9a10b151acd2c28ec4ab2136dba7b2ff1634530bd04b155981a749d08a64?d=mp&s=160"},"body":"On Fri, 2005-10-07 at 17:39, Daniel Barkalow wrote:\n\n> \n> I believe that the situation is the one you describe in your previous \n> message: we determine we need to fetch A and B; we ask for A; we ask for \n> B; we find A isn't available alone, but is available in a pack; we get the \n> pack; we find we now have B (in the pack); the request for B (which would \n> probably fail) is left dangling.\n> \n> The only actual problem I can see is if this happens with a whole bunch of \n> objects at the beginning of a big download, and all but one of your \n> connections are left in this state while you download all of the loose \n> objects over the one connection that got the pack.\n> \n> I don't know if this is a problem for the new http code, but it could be \n> an issue in general if a transport method allocates resources in \n> prefetch().\n\nSo, this sounds like a classic resource scheduling problem\nwith various solutions encoded in many compiler schedulers.\n\nDoesn't this sort of scheduling problem get solved by a\ntwo-stage request pipeline model?  In particular, you only\n\"semi-request A\" and \"semi-request B\" into a queue.  When\nall of the sub-parts (that are needed) of some pack P have\nbeen requested you issue the one request for the common\npack P holding all the sub-parts.  Then all the sub-parts\nare can be retired from the queue.\n\nOr am I just now finally catching up? :-)\n\nThanks,\njdl\n"}]}