{"thread":{"id":"42826","subject":"[RFC 0/3] dumb HTTP transport speedups","startedAt":"2016-07-11T20:51:39Z","lastAt":"2016-07-24T11:20:37Z","messageCount":7,"participants":["Eric Wong","Jakub Narębski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"291210","messageId":"20160711205131.1291-1-e@80x24.org","threadId":"42826","inReplyTo":null,"subject":"[RFC 0/3] dumb HTTP transport speedups","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-11T20:51:28Z","receivedAt":"2016-07-11T20:51:39Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"TL;DR: dumb HTTP clone from a certain badly-packed repo goes from\n~2 hours to ~30 min memory usage drops from 2G to 360M\n\n\nI hadn't packed the public repo at https://public-inbox.org/git\nfor a few weeks.  As an admin of a small server limited memory\nand CPU resources but fairly good bandwidth, I prefer clients\nuse dumb HTTP for initial clones.\n\nUnfortunately, I noticed my dinky netbook runs out-of-memory\nwhen using GIT_SMART_HTTP=0 to clone this giant repo; and a\nmachine with more memory still takes over two hours depending\non network conditions (and uses around 2GB RSS!).\n\nAnyways, https://public-inbox.org/git is better packed, now;\nbut I've kept https://80x24.org/git-i-forgot-to-pack available\nwith over 7K loose objects to illustrate the problem:\n\n\t(this is dumb HTTP-only)\n\tgit clone --mirror https://80x24.org/git-i-forgot-to-pack\n\nThe primary problem is fixed by PATCH 3/3 in this series, and I\ncan now clone the above in around 30 minutes and \"only\" seems to\nuse around 360M memory.\n\nI'll leave git-i-forgot-to-pack up for a few months/year\nso others can test and hammer away at it.\n\nThe following changes since commit 5c589a73de4394ad125a4effac227b3aec856fa1:\n\n  Third batch of topics for 2.10 (2016-07-06 13:42:58 -0700)\n\nare available in the git repository at:\n\n  git://bogomips.org/git-svn.git dumb-speedups\n\nfor you to fetch changes up to b9d5aca4b8e6c9f7fb5ee4e0ce33bb42c4ea2992:\n\n  http-walker: reduce O(n) ops with doubly-linked list (2016-07-11 20:25:51 +0000)\n\n----------------------------------------------------------------\nEric Wong (3):\n      http-walker: remove unused parameter from fetch_object\n      http: avoid disconnecting on 404s for loose objects\n      http-walker: reduce O(n) ops with doubly-linked list\n\n http-walker.c |  55 ++++++++++----------\n http.c        |  16 +++++-\n list.h        | 164 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 204 insertions(+), 31 deletions(-)\n create mode 100644 list.h\n"},{"id":"291211","messageId":"20160711205131.1291-2-e@80x24.org","threadId":"42826","inReplyTo":"20160711205131.1291-1-e@80x24.org","subject":"[PATCH 1/3] http-walker: remove unused parameter from fetch_object","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-11T20:51:29Z","receivedAt":"2016-07-11T20:51:43Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"This parameter has not been used since commit 1d389ab65dc6\n(\"Add support for parallel HTTP transfers\") back in 2005\n\nSigned-off-by: Eric Wong <e@80x24.org>\n---\n http-walker.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/http-walker.c b/http-walker.c\nindex 2c721f0..9f28523 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -447,7 +447,7 @@ static void abort_object_request(struct object_request *obj_req)\n \trelease_object_request(obj_req);\n }\n \n-static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned char *sha1)\n+static int fetch_object(struct walker *walker, unsigned char *sha1)\n {\n \tchar *hex = sha1_to_hex(sha1);\n \tint ret = 0;\n@@ -518,7 +518,7 @@ static int fetch(struct walker *walker, unsigned char *sha1)\n \tstruct walker_data *data = walker->data;\n \tstruct alt_base *altbase = data->alt;\n \n-\tif (!fetch_object(walker, altbase, sha1))\n+\tif (!fetch_object(walker, sha1))\n \t\treturn 0;\n \twhile (altbase) {\n \t\tif (!http_fetch_pack(walker, altbase, sha1))\n-- \nEW\n\n"},{"id":"291212","messageId":"20160711205131.1291-3-e@80x24.org","threadId":"42826","inReplyTo":"20160711205131.1291-1-e@80x24.org","subject":"[PATCH 2/3] http: avoid disconnecting on 404s for loose objects","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-11T20:51:30Z","receivedAt":"2016-07-11T20:51:46Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"404s are common when fetching loose objects on static HTTP\nservers, and reestablishing a connection for every single\n404 adds additional latency.\n\nSigned-off-by: Eric Wong <e@80x24.org>\n---\n http-walker.c |  9 +++++++++\n http.c        | 16 ++++++++++++++--\n 2 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/http-walker.c b/http-walker.c\nindex 9f28523..48f2df4 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -488,6 +488,15 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)\n \t\treq->localfile = -1;\n \t}\n \n+\t/*\n+\t * we turned off CURLOPT_FAILONERROR to avoid losing a\n+\t * persistent connection and got CURLE_OK.\n+\t */\n+\tif (req->http_code == 404 && req->curl_result == CURLE_OK &&\n+\t\t\t(starts_with(req->url, \"http://\") ||\n+\t\t\t starts_with(req->url, \"https://\")))\n+\t\treq->curl_result = CURLE_HTTP_RETURNED_ERROR;\n+\n \tif (obj_req->state == ABORTED) {\n \t\tret = error(\"Request for %s aborted\", hex);\n \t} else if (req->curl_result != CURLE_OK &&\ndiff --git a/http.c b/http.c\nindex d8b2bec..e81dd13 100644\n--- a/http.c\n+++ b/http.c\n@@ -1975,8 +1975,19 @@ static size_t fwrite_sha1_file(char *ptr, size_t eltsize, size_t nmemb,\n \tunsigned char expn[4096];\n \tsize_t size = eltsize * nmemb;\n \tint posn = 0;\n-\tstruct http_object_request *freq =\n-\t\t(struct http_object_request *)data;\n+\tstruct http_object_request *freq = data;\n+\tstruct active_request_slot *slot = freq->slot;\n+\n+\tif (slot) {\n+\t\tCURLcode c = curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CODE,\n+\t\t\t\t\t\t&slot->http_code);\n+\t\tif (c != CURLE_OK)\n+\t\t\tdie(\"BUG: curl_easy_getinfo for HTTP code failed: %s\",\n+\t\t\t\tcurl_easy_strerror(c));\n+\t\tif (slot->http_code >= 400)\n+\t\t\treturn size;\n+\t}\n+\n \tdo {\n \t\tssize_t retval = xwrite(freq->localfile,\n \t\t\t\t\t(char *) ptr + posn, size - posn);\n@@ -2097,6 +2108,7 @@ struct http_object_request *new_http_object_request(const char *base_url,\n \tfreq->slot = get_active_slot();\n \n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_FILE, freq);\n+\tcurl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);\n-- \nEW\n\n"},{"id":"291213","messageId":"20160711205131.1291-4-e@80x24.org","threadId":"42826","inReplyTo":"20160711205131.1291-1-e@80x24.org","subject":"[PATCH 3/3] http-walker: reduce O(n) ops with doubly-linked list","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-11T20:51:31Z","receivedAt":"2016-07-11T20:51:51Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Using the a Linux-kernel-derived doubly-linked list\nimplementation from the Userspace RCU library allows us to\nenqueue and delete items from the object request queue in\nconstant time.\n\nThis change reduces enqueue times in the prefetch() function\nwhere object request queue could grow to several thousand\nobjects.\n\nI left out the list_for_each_entry* family macros from list.h\nwhich relied on the __typeof__ operator as we support platforms\nwithout it.  Thus, list_entry (aka \"container_of\") needs to be\ncalled explicitly inside macro-wrapped for loops.\n\nThe downside is this costs us an additional pointer per object\nrequest, but this is offset by reduced overhead on queue\noperations leading to improved performance and shorter queue\ndepths.\n\nSigned-off-by: Eric Wong <e@80x24.org>\n---\n http-walker.c |  42 ++++++---------\n list.h        | 164 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 179 insertions(+), 27 deletions(-)\n create mode 100644 list.h\n\ndiff --git a/http-walker.c b/http-walker.c\nindex 48f2df4..0b24255 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -2,6 +2,7 @@\n #include \"commit.h\"\n #include \"walker.h\"\n #include \"http.h\"\n+#include \"list.h\"\n \n struct alt_base {\n \tchar *base;\n@@ -23,7 +24,7 @@ struct object_request {\n \tstruct alt_base *repo;\n \tenum object_request_state state;\n \tstruct http_object_request *req;\n-\tstruct object_request *next;\n+\tstruct list_head node;\n };\n \n struct alternates_request {\n@@ -41,7 +42,7 @@ struct walker_data {\n \tstruct alt_base *alt;\n };\n \n-static struct object_request *object_queue_head;\n+static LIST_HEAD(object_queue_head);\n \n static void fetch_alternates(struct walker *walker, const char *base);\n \n@@ -110,19 +111,10 @@ static void process_object_response(void *callback_data)\n \n static void release_object_request(struct object_request *obj_req)\n {\n-\tstruct object_request *entry = object_queue_head;\n-\n \tif (obj_req->req !=NULL && obj_req->req->localfile != -1)\n \t\terror(\"fd leakage in release: %d\", obj_req->req->localfile);\n-\tif (obj_req == object_queue_head) {\n-\t\tobject_queue_head = obj_req->next;\n-\t} else {\n-\t\twhile (entry->next != NULL && entry->next != obj_req)\n-\t\t\tentry = entry->next;\n-\t\tif (entry->next == obj_req)\n-\t\t\tentry->next = entry->next->next;\n-\t}\n \n+\tlist_del(&obj_req->node);\n \tfree(obj_req);\n }\n \n@@ -130,8 +122,10 @@ static void release_object_request(struct object_request *obj_req)\n static int fill_active_slot(struct walker *walker)\n {\n \tstruct object_request *obj_req;\n+\tstruct list_head *pos, *tmp, *head = &object_queue_head;\n \n-\tfor (obj_req = object_queue_head; obj_req; obj_req = obj_req->next) {\n+\tlist_for_each_safe(pos, tmp, head) {\n+\t\tobj_req = list_entry(pos, struct object_request, node);\n \t\tif (obj_req->state == WAITING) {\n \t\t\tif (has_sha1_file(obj_req->sha1))\n \t\t\t\tobj_req->state = COMPLETE;\n@@ -148,7 +142,6 @@ static int fill_active_slot(struct walker *walker)\n static void prefetch(struct walker *walker, unsigned char *sha1)\n {\n \tstruct object_request *newreq;\n-\tstruct object_request *tail;\n \tstruct walker_data *data = walker->data;\n \n \tnewreq = xmalloc(sizeof(*newreq));\n@@ -157,18 +150,9 @@ static void prefetch(struct walker *walker, unsigned char *sha1)\n \tnewreq->repo = data->alt;\n \tnewreq->state = WAITING;\n \tnewreq->req = NULL;\n-\tnewreq->next = NULL;\n \n \thttp_is_verbose = walker->get_verbosely;\n-\n-\tif (object_queue_head == NULL) {\n-\t\tobject_queue_head = newreq;\n-\t} else {\n-\t\ttail = object_queue_head;\n-\t\twhile (tail->next != NULL)\n-\t\t\ttail = tail->next;\n-\t\ttail->next = newreq;\n-\t}\n+\tlist_add_tail(&newreq->node, &object_queue_head);\n \n #ifdef USE_CURL_MULTI\n \tfill_active_slots();\n@@ -451,11 +435,15 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)\n {\n \tchar *hex = sha1_to_hex(sha1);\n \tint ret = 0;\n-\tstruct object_request *obj_req = object_queue_head;\n+\tstruct object_request *obj_req = NULL;\n \tstruct http_object_request *req;\n+\tstruct list_head *pos, *head = &object_queue_head;\n \n-\twhile (obj_req != NULL && hashcmp(obj_req->sha1, sha1))\n-\t\tobj_req = obj_req->next;\n+\tlist_for_each(pos, head) {\n+\t\tobj_req = list_entry(pos, struct object_request, node);\n+\t\tif (!hashcmp(obj_req->sha1, sha1))\n+\t\t\tbreak;\n+\t}\n \tif (obj_req == NULL)\n \t\treturn error(\"Couldn't find request for %s in the queue\", hex);\n \ndiff --git a/list.h b/list.h\nnew file mode 100644\nindex 0000000..f65edce\n--- /dev/null\n+++ b/list.h\n@@ -0,0 +1,164 @@\n+/*\n+ * Copyright (C) 2002 Free Software Foundation, Inc.\n+ * (originally part of the GNU C Library and Userspace RCU)\n+ * Contributed by Ulrich Drepper <drepper@redhat.com>, 2002.\n+ *\n+ * Copyright (C) 2009 Pierre-Marc Fournier\n+ * Conversion to RCU list.\n+ * Copyright (C) 2010 Mathieu Desnoyers <mathieu.desnoyers@efficios.com>\n+ *\n+ * This library is free software; you can redistribute it and/or\n+ * modify it under the terms of the GNU Lesser General Public\n+ * License as published by the Free Software Foundation; either\n+ * version 2.1 of the License, or (at your option) any later version.\n+ *\n+ * This library is distributed in the hope that it will be useful,\n+ * but WITHOUT ANY WARRANTY; without even the implied warranty of\n+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU\n+ * Lesser General Public License for more details.\n+ *\n+ * You should have received a copy of the GNU Lesser General Public\n+ * License along with this library; if not, see\n+ * <http://www.gnu.org/licenses/>.\n+ */\n+\n+#ifndef LIST_H\n+#define LIST_H\t1\n+\n+/*\n+ * The definitions of this file are adopted from those which can be\n+ * found in the Linux kernel headers to enable people familiar with the\n+ * latter find their way in these sources as well.\n+ */\n+\n+/* Basic type for the double-link list. */\n+struct list_head {\n+\tstruct list_head *next, *prev;\n+};\n+\n+/* Define a variable with the head and tail of the list. */\n+#define LIST_HEAD(name) \\\n+\tstruct list_head name = { &(name), &(name) }\n+\n+/* Initialize a new list head. */\n+#define INIT_LIST_HEAD(ptr) \\\n+\t(ptr)->next = (ptr)->prev = (ptr)\n+\n+#define LIST_HEAD_INIT(name) { &(name), &(name) }\n+\n+/* Add new element at the head of the list. */\n+static inline void list_add(struct list_head *newp, struct list_head *head)\n+{\n+\thead->next->prev = newp;\n+\tnewp->next = head->next;\n+\tnewp->prev = head;\n+\thead->next = newp;\n+}\n+\n+/* Add new element at the tail of the list. */\n+static inline void list_add_tail(struct list_head *newp, struct list_head *head)\n+{\n+\thead->prev->next = newp;\n+\tnewp->next = head;\n+\tnewp->prev = head->prev;\n+\thead->prev = newp;\n+}\n+\n+/* Remove element from list. */\n+static inline void __list_del(struct list_head *prev, struct list_head *next)\n+{\n+\tnext->prev = prev;\n+\tprev->next = next;\n+}\n+\n+/* Remove element from list. */\n+static inline void list_del(struct list_head *elem)\n+{\n+\t__list_del(elem->prev, elem->next);\n+}\n+\n+/* Remove element from list, initializing the element's list pointers. */\n+static inline void list_del_init(struct list_head *elem)\n+{\n+\tlist_del(elem);\n+\tINIT_LIST_HEAD(elem);\n+}\n+\n+/* Delete from list, add to another list as head. */\n+static inline void list_move(struct list_head *elem, struct list_head *head)\n+{\n+\t__list_del(elem->prev, elem->next);\n+\tlist_add(elem, head);\n+}\n+\n+/* Replace an old entry. */\n+static inline void list_replace(struct list_head *old, struct list_head *newp)\n+{\n+\tnewp->next = old->next;\n+\tnewp->prev = old->prev;\n+\tnewp->prev->next = newp;\n+\tnewp->next->prev = newp;\n+}\n+\n+/* Join two lists. */\n+static inline void list_splice(struct list_head *add, struct list_head *head)\n+{\n+\t/* Do nothing if the list which gets added is empty. */\n+\tif (add != add->next) {\n+\t\tadd->next->prev = head;\n+\t\tadd->prev->next = head->next;\n+\t\thead->next->prev = add->prev;\n+\t\thead->next = add->next;\n+\t}\n+}\n+\n+/* Get typed element from list at a given position. */\n+#define list_entry(ptr, type, member) \\\n+\t((type *) ((char *) (ptr) - offsetof(type, member)))\n+\n+/* Get first entry from a list. */\n+#define list_first_entry(ptr, type, member) \\\n+\tlist_entry((ptr)->next, type, member)\n+\n+/* Iterate forward over the elements of the list. */\n+#define list_for_each(pos, head) \\\n+\tfor (pos = (head)->next; pos != (head); pos = pos->next)\n+\n+/*\n+ * Iterate forward over the elements list. The list elements can be\n+ * removed from the list while doing this.\n+ */\n+#define list_for_each_safe(pos, p, head) \\\n+\tfor (pos = (head)->next, p = pos->next; \\\n+\t\tpos != (head); \\\n+\t\tpos = p, p = pos->next)\n+\n+/* Iterate backward over the elements of the list. */\n+#define list_for_each_prev(pos, head) \\\n+\tfor (pos = (head)->prev; pos != (head); pos = pos->prev)\n+\n+/*\n+ * Iterate backwards over the elements list. The list elements can be\n+ * removed from the list while doing this.\n+ */\n+#define list_for_each_prev_safe(pos, p, head) \\\n+\tfor (pos = (head)->prev, p = pos->prev; \\\n+\t\tpos != (head); \\\n+\t\tpos = p, p = pos->prev)\n+\n+static inline int list_empty(struct list_head *head)\n+{\n+\treturn head == head->next;\n+}\n+\n+static inline void list_replace_init(struct list_head *old,\n+\t\t\t\t     struct list_head *newp)\n+{\n+\tstruct list_head *head = old->next;\n+\n+\tlist_del(old);\n+\tlist_add_tail(newp, head);\n+\tINIT_LIST_HEAD(old);\n+}\n+\n+#endif /* LIST_H */\n-- \nEW\n\n"},{"id":"291214","messageId":"20160711210243.GA1604@whir","threadId":"42826","inReplyTo":"20160711205131.1291-1-e@80x24.org","subject":"[REJECT 4/3] http-walker: use hashmap to reduce list scan","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-11T21:02:43Z","receivedAt":"2016-07-11T21:02:57Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"For the sake of documentation, I worked on this patch but I\ndon't there was a measurable improvement (hard to tell with\nvariable network conditions) and it increased memory usage to\naround 380M.\n\nI wanted to reduce the list scanning in fill_active_slot() by\ndeleting during iteration, but I'm not sure it helps since the\nloop in that is nowhere near as bad as the prefetch() insertion\nloop fixed in 3/3.\n\nlist_for_each in fetch_object() also tends hit the first object\nin the list when iterating, so there's no improvement I see\nwith this patch.\n\n-----8<------\nSubject: [PATCH] http-walker: use hashmap to reduce list scan\n\nWe can reduce list walking in fill_active_slot by deleting items\nas we walk through the object request queue of pending objects.\n\nHowever, we still need to maintain a mapping of live objects\nfor fetch_object, so introduce the use of a hashmap to keep\ntrack of all live object requests in O(1) average time.\n\nSigned-off-by: Eric Wong <e@80x24.org>\n---\n http-walker.c | 35 +++++++++++++++++++++++++++--------\n 1 file changed, 27 insertions(+), 8 deletions(-)\n\ndiff --git a/http-walker.c b/http-walker.c\nindex 0b24255..8d27707 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -3,6 +3,7 @@\n #include \"walker.h\"\n #include \"http.h\"\n #include \"list.h\"\n+#include \"hashmap.h\"\n \n struct alt_base {\n \tchar *base;\n@@ -19,6 +20,7 @@ enum object_request_state {\n };\n \n struct object_request {\n+\tstruct hashmap_entry ent;\n \tstruct walker *walker;\n \tunsigned char sha1[20];\n \tstruct alt_base *repo;\n@@ -43,6 +45,7 @@ struct walker_data {\n };\n \n static LIST_HEAD(object_queue_head);\n+static struct hashmap *object_requests;\n \n static void fetch_alternates(struct walker *walker, const char *base);\n \n@@ -114,7 +117,11 @@ static void release_object_request(struct object_request *obj_req)\n \tif (obj_req->req !=NULL && obj_req->req->localfile != -1)\n \t\terror(\"fd leakage in release: %d\", obj_req->req->localfile);\n \n-\tlist_del(&obj_req->node);\n+\t/* XXX seems unnecessary with list_del in fill_active_slot */\n+\tif (!list_empty(&obj_req->node))\n+\t\tlist_del(&obj_req->node);\n+\n+\thashmap_remove(object_requests, obj_req, obj_req->sha1);\n \tfree(obj_req);\n }\n \n@@ -127,6 +134,8 @@ static int fill_active_slot(struct walker *walker)\n \tlist_for_each_safe(pos, tmp, head) {\n \t\tobj_req = list_entry(pos, struct object_request, node);\n \t\tif (obj_req->state == WAITING) {\n+\t\t\t/* _init so future list_del is idempotent */\n+\t\t\tlist_del_init(pos);\n \t\t\tif (has_sha1_file(obj_req->sha1))\n \t\t\t\tobj_req->state = COMPLETE;\n \t\t\telse {\n@@ -145,6 +154,8 @@ static void prefetch(struct walker *walker, unsigned char *sha1)\n \tstruct walker_data *data = walker->data;\n \n \tnewreq = xmalloc(sizeof(*newreq));\n+\thashmap_entry_init(&newreq->ent, sha1hash(sha1));\n+\thashmap_add(object_requests, &newreq->ent);\n \tnewreq->walker = walker;\n \thashcpy(newreq->sha1, sha1);\n \tnewreq->repo = data->alt;\n@@ -435,15 +446,12 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)\n {\n \tchar *hex = sha1_to_hex(sha1);\n \tint ret = 0;\n-\tstruct object_request *obj_req = NULL;\n+\tstruct object_request *obj_req;\n+\tstruct hashmap_entry key;\n \tstruct http_object_request *req;\n-\tstruct list_head *pos, *head = &object_queue_head;\n \n-\tlist_for_each(pos, head) {\n-\t\tobj_req = list_entry(pos, struct object_request, node);\n-\t\tif (!hashcmp(obj_req->sha1, sha1))\n-\t\t\tbreak;\n-\t}\n+\thashmap_entry_init(&key, sha1hash(sha1));\n+\tobj_req = hashmap_get(object_requests, &key, sha1);\n \tif (obj_req == NULL)\n \t\treturn error(\"Couldn't find request for %s in the queue\", hex);\n \n@@ -553,6 +561,12 @@ static void cleanup(struct walker *walker)\n \t}\n }\n \n+static int obj_req_cmp(const struct object_request *e1,\n+\t\tconst struct object_request *e2, const unsigned char *sha1)\n+{\n+\treturn hashcmp(e1->sha1, sha1 ? sha1 : e2->sha1);\n+}\n+\n struct walker *get_http_walker(const char *url)\n {\n \tchar *s;\n@@ -580,5 +594,10 @@ struct walker *get_http_walker(const char *url)\n \tadd_fill_function(walker, (int (*)(void *)) fill_active_slot);\n #endif\n \n+\tif (!object_requests) {\n+\t\tobject_requests = xmalloc(sizeof(*object_requests));\n+\t\thashmap_init(object_requests, (hashmap_cmp_fn)obj_req_cmp, 0);\n+\t}\n+\n \treturn walker;\n }\n-- \nEW\n"},{"id":"292052","messageId":"57949467.60904@gmail.com","threadId":"42826","inReplyTo":"20160711205131.1291-1-e@80x24.org","subject":"Re: [RFC 0/3] dumb HTTP transport speedups","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-24T10:11:51Z","receivedAt":"2016-07-24T10:12:08Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-11 o 22:51, Eric Wong pisze:\n\n> TL;DR: dumb HTTP clone from a certain badly-packed repo goes from\n> ~2 hours to ~30 min memory usage drops from 2G to 360M\n> \n> \n> I hadn't packed the public repo at https://public-inbox.org/git\n> for a few weeks.  As an admin of a small server limited memory\n> and CPU resources but fairly good bandwidth, I prefer clients\n> use dumb HTTP for initial clones.\n\nHopefully the solution / workaround for large initial clone\nproblem utilizing bundles (`git bundle`), which can be resumably\ntransferred, would get standarized and automated.\n\nDo you use bitmap indices for speeding up fetches?\n\nBTW. IMVHO the problem with dumb HTTP is the latency, not extra\nbandwidth needed...\n\nBest,\n-- \nJakub Narębski\n\n"},{"id":"292054","messageId":"20160724112027.GB25507@starla","threadId":"42826","inReplyTo":"57949467.60904@gmail.com","subject":"Re: [RFC 0/3] dumb HTTP transport speedups","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-24T11:20:27Z","receivedAt":"2016-07-24T11:20:37Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jakub Narębski <jnareb@gmail.com> wrote:\n> W dniu 2016-07-11 o 22:51, Eric Wong pisze:\n> \n> > TL;DR: dumb HTTP clone from a certain badly-packed repo goes from\n> > ~2 hours to ~30 min memory usage drops from 2G to 360M\n> > \n> > \n> > I hadn't packed the public repo at https://public-inbox.org/git\n> > for a few weeks.  As an admin of a small server limited memory\n> > and CPU resources but fairly good bandwidth, I prefer clients\n> > use dumb HTTP for initial clones.\n> \n> Hopefully the solution / workaround for large initial clone\n> problem utilizing bundles (`git bundle`), which can be resumably\n> transferred, would get standarized and automated.\n\nI've been hoping to look at this more in coming weeks/months.\nIt would be nice if bundles and packs could be unified somehow\nto avoid doubling storage on the server.\n\n> Do you use bitmap indices for speeding up fetches?\n\nYes, but slow clients are still a problem since big responses\nkeeps memory-hungry processes running while trickling\n(or waste disk space buffering the pack output up front)\n\nStatic packfiles/bundles are nice since all the clients can\nshare the same data on the server side as it's trickled out.\n\n> BTW. IMVHO the problem with dumb HTTP is the latency, not extra\n> bandwidth needed...\n\nI enabled persistent connections for 404s on loose objects for\nthis reason :)  We should probably be doing it across the board\non 404s, just haven't gotten around to it...\n\nIncreasing default parallelism should also help; but might hurt\nsome servers which can't handle many connections...\nHard to imagine people using antiquated prefork servers for\nslow clients in a post-Slowloris world, but maybe it happens?\n"}]}