{"thread":{"id":"28903","subject":"[PATCH 0/14] resumable network bundles","startedAt":"2011-11-10T07:43:31Z","lastAt":"2011-11-12T17:58:26Z","messageCount":23,"participants":["Jeff King","Daniel Stenberg","Junio C Hamano","David Michael Barr","Tay Ray Chuan"],"isPatch":true,"patchVersion":1,"patchTotal":14},"messages":[{"id":"179244","messageId":"20111110074330.GA27925@sigill.intra.peff.net","threadId":"28903","inReplyTo":null,"subject":"[PATCH 0/14] resumable network bundles","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:43:31Z","receivedAt":"2011-11-10T07:43:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"One possible option for resumable clones that has been discussed is\nletting the server point the client by http to a static bundle\ncontaining most of history, followed by a fetch from the actual git repo\n(which should be much cheaper now that we have all of the bundled\nhistory).  This series implements \"step 0\" of this plan: just letting\nbundles be fetched across the network in the first place.\n\nShawn raised some issues about using bundles for this (as opposed to\naccessing the packfiles themselves); specifically, this raises the I/O\nfootprint of a repository that has to serve both the bundled version of\nthe pack and the regular packfile.\n\nSo it may be that we don't follow this plan all the way through.\nHowever, even if we don't, fetching bundles over http is still a useful\nthing to be able to do. Which makes this first step worth doing either\nway.\n\n  [01/14]: t/lib-httpd: check for NO_CURL\n  [02/14]: http: turn off curl signals\n  [03/14]: http: refactor http_request function\n  [04/14]: http: add a public function for arbitrary-callback request\n  [05/14]: remote-curl: use http callback for requesting refs\n  [06/14]: transport: factor out bundle to ref list conversion\n  [07/14]: bundle: add is_bundle_buf helper\n  [08/14]: remote-curl: free \"discovery\" object\n  [09/14]: remote-curl: auto-detect bundles when fetching refs\n  [10/14]: remote-curl: try base $URL after $URL/info/refs\n  [11/14]: progress: allow pure-throughput progress meters\n  [12/14]: remote-curl: show progress for bundle downloads\n  [13/14]: remote-curl: resume interrupted bundle transfers\n  [14/14]: clone: give advice on how to resume a failed clone\n\n-Peff\n"},{"id":"179245","messageId":"20111110074527.GA27938@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"Re: [PATCH 0/14] resumable network bundles","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:45:27Z","receivedAt":"2011-11-10T07:45:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 10, 2011 at 02:43:30AM -0500, Jeff King wrote:\n\n>   [01/14]: t/lib-httpd: check for NO_CURL\n>   [02/14]: http: turn off curl signals\n>   [03/14]: http: refactor http_request function\n>   [04/14]: http: add a public function for arbitrary-callback request\n>   [05/14]: remote-curl: use http callback for requesting refs\n>   [06/14]: transport: factor out bundle to ref list conversion\n>   [07/14]: bundle: add is_bundle_buf helper\n>   [08/14]: remote-curl: free \"discovery\" object\n>   [09/14]: remote-curl: auto-detect bundles when fetching refs\n>   [10/14]: remote-curl: try base $URL after $URL/info/refs\n>   [11/14]: progress: allow pure-throughput progress meters\n>   [12/14]: remote-curl: show progress for bundle downloads\n>   [13/14]: remote-curl: resume interrupted bundle transfers\n>   [14/14]: clone: give advice on how to resume a failed clone\n\nI forgot to mention: this goes on top of mf/curl-select-fdset. It's only\nin next now, but some of my http cleanups build semantically on the\ncleanups in that topic.\n\n-Peff\n"},{"id":"179246","messageId":"20111110074600.GA27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 01/14] t/lib-httpd: check for NO_CURL","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:46:00Z","receivedAt":"2011-11-10T07:46:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"We already do this check individually in each of the tests\nthat includes lib-httpd. Let's factor it out.\n\nThere is one test (t5540) that uses lib-httpd but does not\ncurrently do this check. But it actually has a stricter\ncheck which is a superset (it needs all of the requirements\nto have built git-http-push, one of which is not setting\nNO_CURL), so adding this extra check won't hurt anything.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n t/lib-httpd.sh          |    5 +++++\n t/t5541-http-push.sh    |    5 -----\n t/t5550-http-fetch.sh   |    5 -----\n t/t5551-http-fetch.sh   |    5 -----\n t/t5561-http-backend.sh |    5 -----\n 5 files changed, 5 insertions(+), 20 deletions(-)\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex f7dc078..8331527 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -3,6 +3,11 @@\n # Copyright (c) 2008 Clemens Buchacher <drizzd@aon.at>\n #\n \n+if test -n \"$NO_CURL\"; then\n+\tskip_all='skipping test, git built without http support'\n+\ttest_done\n+fi\n+\n if test -z \"$GIT_TEST_HTTPD\"\n then\n \tskip_all=\"Network testing disabled (define GIT_TEST_HTTPD to enable)\"\ndiff --git a/t/t5541-http-push.sh b/t/t5541-http-push.sh\nindex a73c826..a326ee0 100755\n--- a/t/t5541-http-push.sh\n+++ b/t/t5541-http-push.sh\n@@ -6,11 +6,6 @@\n test_description='test smart pushing over http via http-backend'\n . ./test-lib.sh\n \n-if test -n \"$NO_CURL\"; then\n-\tskip_all='skipping test, git built without http support'\n-\ttest_done\n-fi\n-\n ROOT_PATH=\"$PWD\"\n LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5541'}\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\ndiff --git a/t/t5550-http-fetch.sh b/t/t5550-http-fetch.sh\nindex 311a33c..e9282c5 100755\n--- a/t/t5550-http-fetch.sh\n+++ b/t/t5550-http-fetch.sh\n@@ -3,11 +3,6 @@\n test_description='test dumb fetching over http via static file'\n . ./test-lib.sh\n \n-if test -n \"$NO_CURL\"; then\n-\tskip_all='skipping test, git built without http support'\n-\ttest_done\n-fi\n-\n LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5550'}\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\ndiff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh\nindex 26d3557..3557f2e 100755\n--- a/t/t5551-http-fetch.sh\n+++ b/t/t5551-http-fetch.sh\n@@ -3,11 +3,6 @@\n test_description='test smart fetching over http via http-backend'\n . ./test-lib.sh\n \n-if test -n \"$NO_CURL\"; then\n-\tskip_all='skipping test, git built without http support'\n-\ttest_done\n-fi\n-\n LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5551'}\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\ndiff --git a/t/t5561-http-backend.sh b/t/t5561-http-backend.sh\nindex b5d7fbc..974be7c 100755\n--- a/t/t5561-http-backend.sh\n+++ b/t/t5561-http-backend.sh\n@@ -3,11 +3,6 @@\n test_description='test git-http-backend'\n . ./test-lib.sh\n \n-if test -n \"$NO_CURL\"; then\n-\tskip_all='skipping test, git built without http support'\n-\ttest_done\n-fi\n-\n LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5561'}\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179247","messageId":"20111110074803.GB27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 02/14] http: turn off curl signals","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:48:03Z","receivedAt":"2011-11-10T07:48:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Curl sets and clears the handler for SIGALRM, which makes it\nincompatible with git's progress code. However, we can ask\ncurl not to do this.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nI'm a little iffy on this one. If I understand correctly, depending on\nthe build and configuration, curl may not be able to timeout during DNS\nlookups. But I'm not sure if it does, anyway, since we don't set any\ntimeouts.\n\nAn alternate plan would be to give the progress code a mode where it\ngets poked by curl every second or so (curl has a PROGRESSFUNCTION\noption for doing this).\n\n http.c |    2 ++\n 1 files changed, 2 insertions(+), 0 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 95c2dfd..4f9e004 100644\n--- a/http.c\n+++ b/http.c\n@@ -318,6 +318,8 @@ static int has_cert_password(void)\n \tif (curl_http_proxy)\n \t\tcurl_easy_setopt(result, CURLOPT_PROXY, curl_http_proxy);\n \n+\tcurl_easy_setopt(result, CURLOPT_NOSIGNAL, 1);\n+\n \treturn result;\n }\n \n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179248","messageId":"20111110074831.GC27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 03/14] http: refactor http_request function","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:48:31Z","receivedAt":"2011-11-10T07:48:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This function takes a flag to indicate where the output\nshould go (either to a file or to a strbuf). This flag is\nmostly used to set the callback function we hand to curl.\n\nThis isn't very flexible for adding new output types.\nInstead, let's just let callers pass in the callback\nfunction directly. This results in shorter, more readable,\nand more flexible code.\n\nThe only other thing the flag was used for was to set a\n\"Range\" header when we already have a partial file (by using\nthe results of ftell). This patch also adds an \"offset\"\nparameter, which can be used by callers to specify this\nfeature separately (which is also more flexible, as non-FILE\ncallers can now resume partial transfers).\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n http.c |   37 ++++++++++++++-----------------------\n 1 files changed, 14 insertions(+), 23 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 4f9e004..9ffd894 100644\n--- a/http.c\n+++ b/http.c\n@@ -797,11 +797,8 @@ void append_remote_object_url(struct strbuf *buf, const char *url,\n \treturn strbuf_detach(&buf, NULL);\n }\n \n-/* http_request() targets */\n-#define HTTP_REQUEST_STRBUF\t0\n-#define HTTP_REQUEST_FILE\t1\n-\n-static int http_request(const char *url, void *result, int target, int options)\n+static int http_request(const char *url, curl_write_callback cb, void *result,\n+\t\t\tlong offset, int options)\n {\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n@@ -818,19 +815,13 @@ static int http_request(const char *url, void *result, int target, int options)\n \t} else {\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, result);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, cb);\n+\t}\n \n-\t\tif (target == HTTP_REQUEST_FILE) {\n-\t\t\tlong posn = ftell(result);\n-\t\t\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION,\n-\t\t\t\t\t fwrite);\n-\t\t\tif (posn > 0) {\n-\t\t\t\tstrbuf_addf(&buf, \"Range: bytes=%ld-\", posn);\n-\t\t\t\theaders = curl_slist_append(headers, buf.buf);\n-\t\t\t\tstrbuf_reset(&buf);\n-\t\t\t}\n-\t\t} else\n-\t\t\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION,\n-\t\t\t\t\t fwrite_buffer);\n+\tif (offset > 0) {\n+\t\tstrbuf_addf(&buf, \"Range: bytes=%lu-\", offset);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n+\t\tstrbuf_reset(&buf);\n \t}\n \n \tstrbuf_addstr(&buf, \"Pragma:\");\n@@ -881,18 +872,18 @@ static int http_request(const char *url, void *result, int target, int options)\n \treturn ret;\n }\n \n-static int http_request_reauth(const char *url, void *result, int target,\n-\t\t\t       int options)\n+static int http_request_reauth(const char *url, curl_write_callback cb,\n+\t\t\t       void *result, unsigned long offset, int options)\n {\n-\tint ret = http_request(url, result, target, options);\n+\tint ret = http_request(url, cb, result, offset, options);\n \tif (ret != HTTP_REAUTH)\n \t\treturn ret;\n-\treturn http_request(url, result, target, options);\n+\treturn http_request(url, cb, result, offset, options);\n }\n \n int http_get_strbuf(const char *url, struct strbuf *result, int options)\n {\n-\treturn http_request_reauth(url, result, HTTP_REQUEST_STRBUF, options);\n+\treturn http_request_reauth(url, fwrite_buffer, result, 0, options);\n }\n \n /*\n@@ -915,7 +906,7 @@ static int http_get_file(const char *url, const char *filename, int options)\n \t\tgoto cleanup;\n \t}\n \n-\tret = http_request_reauth(url, result, HTTP_REQUEST_FILE, options);\n+\tret = http_request_reauth(url, NULL, result, ftell(result), options);\n \tfclose(result);\n \n \tif ((ret == HTTP_OK) && move_temp_to_file(tmpfile.buf, filename))\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179250","messageId":"20111110074920.GD27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 04/14] http: add a public function for arbitrary-callback request","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:49:20Z","receivedAt":"2011-11-10T07:49:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"The http_request function recently learned to take arbitrary\ncallbacks; let's expose this functionality to callers.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n http.c |    6 ++++++\n http.h |    3 +++\n 2 files changed, 9 insertions(+), 0 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 9ffd894..91451e9 100644\n--- a/http.c\n+++ b/http.c\n@@ -886,6 +886,12 @@ int http_get_strbuf(const char *url, struct strbuf *result, int options)\n \treturn http_request_reauth(url, fwrite_buffer, result, 0, options);\n }\n \n+int http_get_callback(const char *url, curl_write_callback cb,\n+\t\t      void *data, long offset, int options)\n+{\n+\treturn http_request_reauth(url, cb, data, offset, options);\n+}\n+\n /*\n  * Downloads an url and stores the result in the given file.\n  *\ndiff --git a/http.h b/http.h\nindex ee16069..4977bde 100644\n--- a/http.h\n+++ b/http.h\n@@ -132,6 +132,9 @@ extern void append_remote_object_url(struct strbuf *buf, const char *url,\n  */\n int http_get_strbuf(const char *url, struct strbuf *result, int options);\n \n+int http_get_callback(const char *url, curl_write_callback cb, void *data,\n+\t\t      long offset, int options);\n+\n /*\n  * Prints an error message using error() containing url and curl_errorstr,\n  * and returns ret.\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179251","messageId":"20111110074944.GE27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 05/14] remote-curl: use http callback for requesting refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:49:44Z","receivedAt":"2011-11-10T07:49:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This should behave identically to the current strbuf code,\nbut opens up room for us to do more clever things with\nbundles in a future patch.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nObviously it's way more code for the same thing, but future patches will\nmake the design more clear.\n\n remote-curl.c |   22 ++++++++++++++++++++--\n 1 files changed, 20 insertions(+), 2 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 0e720ee..fb4d853 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -90,6 +90,24 @@ static void free_discovery(struct discovery *d)\n \t}\n }\n \n+struct get_refs_cb_data {\n+\tstruct strbuf *out;\n+};\n+\n+static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)\n+{\n+\tstruct get_refs_cb_data *data = vdata;\n+\tstrbuf_add(data->out, buf, sz * n);\n+\treturn sz * n;\n+}\n+\n+static int get_refs_from_url(const char *url, struct strbuf *out, int options)\n+{\n+\tstruct get_refs_cb_data data;\n+\tdata.out = out;\n+\treturn http_get_callback(url, get_refs_callback, &data, 0, options);\n+}\n+\n static struct discovery* discover_refs(const char *service)\n {\n \tstruct strbuf buffer = STRBUF_INIT;\n@@ -112,7 +130,7 @@ static void free_discovery(struct discovery *d)\n \t}\n \trefs_url = strbuf_detach(&buffer, NULL);\n \n-\thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n+\thttp_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE);\n \n \t/* try again with \"plain\" url (no ? or & appended) */\n \tif (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {\n@@ -123,7 +141,7 @@ static void free_discovery(struct discovery *d)\n \t\tstrbuf_addf(&buffer, \"%sinfo/refs\", url);\n \t\trefs_url = strbuf_detach(&buffer, NULL);\n \n-\t\thttp_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);\n+\t\thttp_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE);\n \t}\n \n \tswitch (http_ret) {\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179252","messageId":"20111110074952.GF27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 06/14] transport: factor out bundle to ref list conversion","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:49:52Z","receivedAt":"2011-11-10T07:49:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"There is a data structure mismatch between what the\ntransport code wants (a linked list of \"struct ref\") and\nwhat the bundle header provides (an array of ref names and\nsha1s), so the transport code has to convert.\n\nLet's factor out this conversion to make it useful to other\ntransport-ish callers (like remote-curl).\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n bundle.c    |   16 ++++++++++++++++\n bundle.h    |    2 ++\n transport.c |   11 +----------\n 3 files changed, 19 insertions(+), 10 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 08020bc..e48fe2f 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -7,6 +7,7 @@\n #include \"list-objects.h\"\n #include \"run-command.h\"\n #include \"refs.h\"\n+#include \"remote.h\"\n \n static const char bundle_signature[] = \"# v2 git bundle\\n\";\n \n@@ -449,3 +450,18 @@ int unbundle(struct bundle_header *header, int bundle_fd, int flags)\n \t\treturn error(\"index-pack died\");\n \treturn 0;\n }\n+\n+struct ref *bundle_header_to_refs(const struct bundle_header *header)\n+{\n+\tstruct ref *result = NULL;\n+\tint i;\n+\n+\tfor (i = 0; i < header->references.nr; i++) {\n+\t\tstruct ref_list_entry *e = header->references.list + i;\n+\t\tstruct ref *ref = alloc_ref(e->name);\n+\t\thashcpy(ref->old_sha1, e->sha1);\n+\t\tref->next = result;\n+\t\tresult = ref;\n+\t}\n+\treturn result;\n+}\ndiff --git a/bundle.h b/bundle.h\nindex 1584e4d..675cc97 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -24,4 +24,6 @@ int create_bundle(struct bundle_header *header, const char *path,\n int list_bundle_refs(struct bundle_header *header,\n \t\tint argc, const char **argv);\n \n+struct ref *bundle_header_to_refs(const struct bundle_header *header);\n+\n #endif\ndiff --git a/transport.c b/transport.c\nindex 51814b5..5020bbb 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -407,8 +407,6 @@ struct bundle_transport_data {\n static struct ref *get_refs_from_bundle(struct transport *transport, int for_push)\n {\n \tstruct bundle_transport_data *data = transport->data;\n-\tstruct ref *result = NULL;\n-\tint i;\n \n \tif (for_push)\n \t\treturn NULL;\n@@ -418,14 +416,7 @@ struct bundle_transport_data {\n \tdata->fd = read_bundle_header(transport->url, &data->header);\n \tif (data->fd < 0)\n \t\tdie (\"Could not read bundle '%s'.\", transport->url);\n-\tfor (i = 0; i < data->header.references.nr; i++) {\n-\t\tstruct ref_list_entry *e = data->header.references.list + i;\n-\t\tstruct ref *ref = alloc_ref(e->name);\n-\t\thashcpy(ref->old_sha1, e->sha1);\n-\t\tref->next = result;\n-\t\tresult = ref;\n-\t}\n-\treturn result;\n+\treturn bundle_header_to_refs(&data->header);\n }\n \n static int fetch_refs_from_bundle(struct transport *transport,\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179253","messageId":"20111110075001.GG27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 07/14] bundle: add is_bundle_buf helper","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:50:01Z","receivedAt":"2011-11-10T07:50:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This is similar to is_bundle, but checks an in-memory buffer\nrather than a file. It also works on a partial buffer,\nchecking only the bundle signature. This means the result is\na three-way conditional: yes, no, or \"we do not have enough\ndata yet\".\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n bundle.c |   14 ++++++++++++++\n bundle.h |    1 +\n 2 files changed, 15 insertions(+), 0 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex e48fe2f..6f911bc 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -122,6 +122,20 @@ int is_bundle(const char *path, int quiet)\n \treturn (fd >= 0);\n }\n \n+int is_bundle_buf(const char *s, int len)\n+{\n+\tif (len > strlen(bundle_signature))\n+\t\tlen = strlen(bundle_signature);\n+\t/* If we don't match what we already have, then definitely not. */\n+\tif (memcmp(s, bundle_signature, len))\n+\t\treturn 0;\n+\t/* If we have enough bytes, we can say yes */\n+\tif (len == strlen(bundle_signature))\n+\t\treturn 1;\n+\t/* otherwise, we can only say \"maybe\" */\n+\treturn -1;\n+}\n+\n static int list_refs(struct ref_list *r, int argc, const char **argv)\n {\n \tint i;\ndiff --git a/bundle.h b/bundle.h\nindex 675cc97..8bec44d 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -15,6 +15,7 @@ struct bundle_header {\n };\n \n int is_bundle(const char *path, int quiet);\n+int is_bundle_buf(const char *s, int len);\n int read_bundle_header(const char *path, struct bundle_header *header);\n int create_bundle(struct bundle_header *header, const char *path,\n \t\tint argc, const char **argv);\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179254","messageId":"20111110075008.GH27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 08/14] remote-curl: free \"discovery\" object","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:50:08Z","receivedAt":"2011-11-10T07:50:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"We cache the last-used \"discovery\" object, which contains\nthe data we pulled from the remote about which refs it has,\nwhich saves us an HTTP round-trip when somebody does\nsomething like \"list\" followed by \"fetch\".\n\nWe don't bother free()ing it at the end of the program\nbecause it just contains memory which will be reclaimed by\nthe OS. However, cleaning up explicitly will future-proof us\nagainst later changes which will add external storage (like\ntemporary files).\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n remote-curl.c |    1 +\n 1 files changed, 1 insertions(+), 0 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex fb4d853..014d413 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -933,6 +933,7 @@ int main(int argc, const char **argv)\n \t\tstrbuf_reset(&buf);\n \t} while (1);\n \n+\tfree_discovery(last_discovery);\n \thttp_cleanup();\n \n \treturn 0;\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179255","messageId":"20111110075052.GI27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 09/14] remote-curl: auto-detect bundles when fetching refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:50:52Z","receivedAt":"2011-11-10T07:50:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"You can't currently fetch from a network bundle, like:\n\n  git fetch http://example.com/foo.bundle\n\nThis patch takes the first (and biggest) step towards that\nworking: it auto-detects when fetching refs results in a\nbundle, and automatically spools the bundle to disk and\nfetches from it.\n\nThere are a few important design decisions to note:\n\n  1. We auto-detect the bundle based on content, not based\n     on a special token in the URL (like ending in\n     \".bundle\"). This lets the server side be flexible with\n     its URLs (e.g., \"http://example.com/bundle?repo=foo\").\n\n  2. When fetching refs, we don't actually fetch $URL, but\n     start with $URL/info/refs, looking for smart or dumb\n     http. Some servers, when file \"foo.bundle\" exists, will\n     serve it to the client when \"foo.bundle/info/refs\" is\n     requested. Therefore we may be \"surprised\" to receive a\n     bundle when we thought we were just getting the list of\n     refs, and need to handle it appropriately.\n\n  3. We spool the bundle to disk, and then run \"index-pack\n     --fix-thin\" to create a packfile. That means we will\n     momentarily use twice the size of the bundle in local\n     disk space. Avoiding this would mean piping directly to\n     \"index-pack --fix-thin\".  However, if we want to be\n     able to resume the transfer of the bundle after an\n     interruption, then we need to save the bundle's pack.\n\n     In theory a smart index-pack that was interrupted could\n     write out its partial results along with a count of how\n     many bytes it actually consumed (i.e., where to resume\n     next time), and then pick up where it left off when fed\n     the rest of the data. But index-pack isn't that smart\n     yet, so let's start off with spooling.\n\nNo tests yet, as apache is not one of the \"surprising\"\nservers from (2), and our test harness is based around that\n(though just with this patch, you can fetch from surprising\nservers like lighttpd).\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nThis is really the big, interesting one.\n\n remote-curl.c |  124 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 119 insertions(+), 5 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 014d413..84586e0 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -7,6 +7,7 @@\n #include \"run-command.h\"\n #include \"pkt-line.h\"\n #include \"sideband.h\"\n+#include \"bundle.h\"\n \n static struct remote *remote;\n static const char *url; /* always ends with a trailing slash */\n@@ -77,6 +78,10 @@ struct discovery {\n \tchar *buf;\n \tsize_t len;\n \tunsigned proto_git : 1;\n+\n+\tchar *bundle_filename;\n+\tint bundle_fd;\n+\tstruct bundle_header bundle_header;\n };\n static struct discovery *last_discovery;\n \n@@ -86,26 +91,93 @@ static void free_discovery(struct discovery *d)\n \t\tif (d == last_discovery)\n \t\t\tlast_discovery = NULL;\n \t\tfree(d->buf_alloc);\n+\t\tif (d->bundle_fd >= 0)\n+\t\t\tclose(d->bundle_fd);\n+\t\tif (d->bundle_filename) {\n+\t\t\tunlink(d->bundle_filename);\n+\t\t\tfree(d->bundle_filename);\n+\t\t}\n \t\tfree(d);\n \t}\n }\n \n struct get_refs_cb_data {\n \tstruct strbuf *out;\n+\n+\tint is_bundle;\n+\tconst char *tmpname;\n+\tFILE *fh;\n };\n \n static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)\n {\n \tstruct get_refs_cb_data *data = vdata;\n-\tstrbuf_add(data->out, buf, sz * n);\n+\tstruct strbuf *out = data->out;\n+\n+\tif (data->is_bundle > 0)\n+\t\treturn fwrite(buf, sz, n, data->fh);\n+\n+\tstrbuf_add(out, buf, sz * n);\n+\n+\tif (data->is_bundle == 0)\n+\t\treturn sz * n;\n+\n+\tdata->is_bundle = is_bundle_buf(out->buf, out->len);\n+\tif (data->is_bundle > 0) {\n+\t\tdata->fh = fopen(data->tmpname, \"wb\");\n+\t\tif (!data->fh)\n+\t\t\tdie_errno(\"unable to open %s\", data->tmpname);\n+\t\tif (fwrite(out->buf, 1, out->len, data->fh) < out->len)\n+\t\t\tdie_errno(\"unable to write to %s\", data->tmpname);\n+\t}\n \treturn sz * n;\n }\n \n-static int get_refs_from_url(const char *url, struct strbuf *out, int options)\n+static int get_refs_from_url(const char *url, struct strbuf *out, int options,\n+\t\t\t     const char *tmpname, int *is_bundle)\n {\n \tstruct get_refs_cb_data data;\n+\tint ret;\n+\n \tdata.out = out;\n-\treturn http_get_callback(url, get_refs_callback, &data, 0, options);\n+\tdata.is_bundle = -1;\n+\tdata.tmpname = tmpname;\n+\tdata.fh = NULL;\n+\n+\tret = http_get_callback(url, get_refs_callback, &data, 0, options);\n+\n+\tif (data.fh) {\n+\t\tif (fclose(data.fh))\n+\t\t\tdie_errno(\"unable to write to %s\", data.tmpname);\n+\t}\n+\n+\t*is_bundle = data.is_bundle > 0;\n+\treturn ret;\n+}\n+\n+static const char *url_to_bundle_tmpfile(const char *url)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tint last_was_quoted = 1;\n+\tconst char *ret;\n+\n+\tstrbuf_addstr(&buf, \"tmp_bundle_\");\n+\tfor (; *url; url++) {\n+\t\tif (isalpha(*url) || isdigit(*url)) {\n+\t\t\tstrbuf_addch(&buf, *url);\n+\t\t\tlast_was_quoted = 0;\n+\t\t}\n+\t\telse if (!last_was_quoted) {\n+\t\t\tstrbuf_addch(&buf, '_');\n+\t\t\tlast_was_quoted = 1;\n+\t\t}\n+\t}\n+\tif (last_was_quoted)\n+\t\tstrbuf_setlen(&buf, buf.len - 1);\n+\n+\tret = git_path(\"objects/%s\", buf.buf);\n+\tstrbuf_release(&buf);\n+\treturn ret;\n }\n \n static struct discovery* discover_refs(const char *service)\n@@ -114,11 +186,15 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)\n \tstruct discovery *last = last_discovery;\n \tchar *refs_url;\n \tint http_ret, is_http = 0, proto_git_candidate = 1;\n+\tconst char *filename;\n+\tint is_bundle;\n \n \tif (last && !strcmp(service, last->service))\n \t\treturn last;\n \tfree_discovery(last);\n \n+\tfilename = url_to_bundle_tmpfile(url);\n+\n \tstrbuf_addf(&buffer, \"%sinfo/refs\", url);\n \tif (!prefixcmp(url, \"http://\") || !prefixcmp(url, \"https://\")) {\n \t\tis_http = 1;\n@@ -130,7 +206,8 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)\n \t}\n \trefs_url = strbuf_detach(&buffer, NULL);\n \n-\thttp_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE);\n+\thttp_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE,\n+\t\t\t\t     filename, &is_bundle);\n \n \t/* try again with \"plain\" url (no ? or & appended) */\n \tif (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {\n@@ -141,7 +218,8 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)\n \t\tstrbuf_addf(&buffer, \"%sinfo/refs\", url);\n \t\trefs_url = strbuf_detach(&buffer, NULL);\n \n-\t\thttp_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE);\n+\t\thttp_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE,\n+\t\t\t\t\t     filename, &is_bundle);\n \t}\n \n \tswitch (http_ret) {\n@@ -161,6 +239,7 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)\n \tlast->service = service;\n \tlast->buf_alloc = strbuf_detach(&buffer, &last->len);\n \tlast->buf = last->buf_alloc;\n+\tlast->bundle_fd = -1;\n \n \tif (is_http && proto_git_candidate\n \t\t&& 5 <= last->len && last->buf[4] == '#') {\n@@ -190,6 +269,10 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)\n \t\tlast->proto_git = 1;\n \t}\n \n+\telse if (is_bundle) {\n+\t\tlast->bundle_filename = xstrdup(filename);\n+\t}\n+\n \tfree(refs_url);\n \tstrbuf_release(&buffer);\n \tlast_discovery = last;\n@@ -276,6 +359,22 @@ static int write_discovery(int in, int out, void *data)\n \treturn refs;\n }\n \n+static void ensure_bundle_open(struct discovery *heads)\n+{\n+\tif (heads->bundle_fd >= 0)\n+\t\treturn;\n+\theads->bundle_fd = read_bundle_header(heads->bundle_filename,\n+\t\t\t\t\t      &heads->bundle_header);\n+\tif (heads->bundle_fd < 0)\n+\t\tdie(\"could not read bundle from %s\", url);\n+}\n+\n+static struct ref *parse_bundle_refs(struct discovery *heads)\n+{\n+\tensure_bundle_open(heads);\n+\treturn bundle_header_to_refs(&heads->bundle_header);\n+}\n+\n static struct ref *get_refs(int for_push)\n {\n \tstruct discovery *heads;\n@@ -287,6 +386,11 @@ static int write_discovery(int in, int out, void *data)\n \n \tif (heads->proto_git)\n \t\treturn parse_git_refs(heads);\n+\tif (heads->bundle_filename) {\n+\t\tif (for_push)\n+\t\t\tdie(\"cannot push into a remote bundle\");\n+\t\treturn parse_bundle_refs(heads);\n+\t}\n \treturn parse_info_refs(heads);\n }\n \n@@ -690,11 +794,21 @@ static int fetch_git(struct discovery *heads,\n \treturn err;\n }\n \n+static int fetch_bundle(struct discovery *d,\n+\t\t\tint nr_heads, struct ref **to_fetch)\n+{\n+\tensure_bundle_open(d);\n+\treturn unbundle(&d->bundle_header, d->bundle_fd,\n+\t\t\toptions.progress ? BUNDLE_VERBOSE : 0);\n+}\n+\n static int fetch(int nr_heads, struct ref **to_fetch)\n {\n \tstruct discovery *d = discover_refs(\"git-upload-pack\");\n \tif (d->proto_git)\n \t\treturn fetch_git(d, nr_heads, to_fetch);\n+\telse if (d->bundle_filename)\n+\t\treturn fetch_bundle(d, nr_heads, to_fetch);\n \telse\n \t\treturn fetch_dumb(nr_heads, to_fetch);\n }\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179256","messageId":"20111110075118.GJ27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 10/14] remote-curl: try base $URL after $URL/info/refs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:51:18Z","receivedAt":"2011-11-10T07:51:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"When fetching via http, we will try \"$URL/info/refs\" to get\nthe list of refs. We may get an unexpected bundle from that\ntransfer, and we already handle that case. But we should\nalso check just \"$URL\" to see if it's a bundle.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nAnd now we can actually test with apache.\n\n remote-curl.c          |   19 +++++++++++++++++++\n t/t5552-http-bundle.sh |   36 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 55 insertions(+), 0 deletions(-)\n create mode 100755 t/t5552-http-bundle.sh\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 84586e0..7734495 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -222,6 +222,25 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,\n \t\t\t\t\t     filename, &is_bundle);\n \t}\n \n+\t/* try the straight URL for a bundle, but don't impact the\n+\t * error reporting that happens below. */\n+\tif (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {\n+\t\tstruct strbuf trimmed = STRBUF_INIT;\n+\t\tint r;\n+\n+\t\tstrbuf_reset(&buffer);\n+\n+\t\tstrbuf_addstr(&trimmed, url);\n+\t\twhile (trimmed.len > 0 && trimmed.buf[trimmed.len-1] == '/')\n+\t\t\tstrbuf_setlen(&trimmed, trimmed.len - 1);\n+\n+\t\tr = get_refs_from_url(trimmed.buf, &buffer, 0, filename,\n+\t\t\t\t      &is_bundle);\n+\t\tif (r == HTTP_OK && is_bundle)\n+\t\t\thttp_ret = r;\n+\t\tstrbuf_release(&trimmed);\n+\t}\n+\n \tswitch (http_ret) {\n \tcase HTTP_OK:\n \t\tbreak;\ndiff --git a/t/t5552-http-bundle.sh b/t/t5552-http-bundle.sh\nnew file mode 100755\nindex 0000000..8527403\n--- /dev/null\n+++ b/t/t5552-http-bundle.sh\n@@ -0,0 +1,36 @@\n+#!/bin/sh\n+\n+test_description='test fetching from http-accessible bundles'\n+. ./test-lib.sh\n+\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5552'}\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create bundles' '\n+\ttest_commit one &&\n+\tgit bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/one.bundle\" --all &&\n+\ttest_commit two &&\n+\tgit bundle create \"$HTTPD_DOCUMENT_ROOT_PATH/two.bundle\" --all ^one\n+'\n+\n+test_expect_success 'clone from bundle' '\n+\tgit clone --bare $HTTPD_URL/one.bundle clone &&\n+\techo one >expect &&\n+\tgit --git-dir=clone log -1 --format=%s >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'fetch from bundle' '\n+\tgit --git-dir=clone fetch $HTTPD_URL/two.bundle refs/*:refs/* &&\n+\techo two >expect &&\n+\tgit --git-dir=clone log -1 --format=%s >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'cannot clone from partial bundle' '\n+\ttest_must_fail git clone $HTTPD_URL/two.bundle\n+'\n+\n+stop_httpd\n+test_done\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179257","messageId":"20111110075300.GK27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 11/14] progress: allow pure-throughput progress meters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:53:00Z","receivedAt":"2011-11-10T07:53:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"The progress code assumes we are counting something (usually\nobjects), even if we are measuring throughput. This works\nfor fetching packfiles, since they show us the object count\nalongside the throughput, like:\n\n  Receiving objects:   2% (301/11968), 22.00 MiB | 10.97 MiB/s\n\nYou can also tell the progress code you don't know how many\nitems you have (by specifying a total of 0), and it looks\nlike:\n\n  Counting objects: 34957\n\nHowever, if you're fetching a single large item, you want\nthroughput but you might not have a meaningful count. You\ncan say you are getting item 0 or 1 out of 1 total, but then\nthe percent meter is misleading:\n\n  Downloading:   0% (0/1), 22.00 MiB | 10.97 MiB/s\n\nor\n\n  Downloading: 100% (0/1), 22.00 MiB | 10.97 MiB/s\n\nNeither of those is accurate. You are probably somewhere\nbetween zero and 100 percent through the operation, but you\ndon't know how far.\n\nTelling it you don't know how many items is even uglier:\n\n  Downloading: 1, 22.00 MiB | 10.97 MiB/s\n\nInstead, this patch will omit the count entirely if you are\non the zero-th item of an unknown number of items. It looks\nlike:\n\n  Downloading: 22.00 MiB | 10.97 MiB/s\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nThis was the last amount of work to massage the progress code into doing\nwhat I wanted. It might be nicer if it could show a percentage (if we\nknow the total size), but there's even more surgery required for that.\n\n progress.c |   20 ++++++++++++--------\n 1 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 3971f49..92fc3c5 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -103,7 +103,10 @@ static int display(struct progress *progress, unsigned n, const char *done)\n \t\t\treturn 1;\n \t\t}\n \t} else if (progress_update) {\n-\t\tfprintf(stderr, \"%s: %u%s%s\", progress->title, n, tp, eol);\n+\t\tfprintf(stderr, \"%s: \", progress->title);\n+\t\tif (n)\n+\t\t\tfprintf(stderr, \"%u\", n);\n+\t\tfprintf(stderr, \"%s%s\", tp, eol);\n \t\tfflush(stderr);\n \t\tprogress_update = 0;\n \t\treturn 1;\n@@ -113,23 +116,24 @@ static int display(struct progress *progress, unsigned n, const char *done)\n }\n \n static void throughput_string(struct throughput *tp, off_t total,\n-\t\t\t      unsigned int rate)\n+\t\t\t      unsigned int rate, struct progress *p)\n {\n+\tconst char *delim = p->total || p->last_value > 0 ? \", \" : \"\";\n \tint l = sizeof(tp->display);\n \tif (total > 1 << 30) {\n-\t\tl -= snprintf(tp->display, l, \", %u.%2.2u GiB\",\n+\t\tl -= snprintf(tp->display, l, \"%s%u.%2.2u GiB\", delim,\n \t\t\t      (int)(total >> 30),\n \t\t\t      (int)(total & ((1 << 30) - 1)) / 10737419);\n \t} else if (total > 1 << 20) {\n \t\tint x = total + 5243;  /* for rounding */\n-\t\tl -= snprintf(tp->display, l, \", %u.%2.2u MiB\",\n+\t\tl -= snprintf(tp->display, l, \"%s%u.%2.2u MiB\", delim,\n \t\t\t      x >> 20, ((x & ((1 << 20) - 1)) * 100) >> 20);\n \t} else if (total > 1 << 10) {\n \t\tint x = total + 5;  /* for rounding */\n-\t\tl -= snprintf(tp->display, l, \", %u.%2.2u KiB\",\n+\t\tl -= snprintf(tp->display, l, \"%s%u.%2.2u KiB\", delim,\n \t\t\t      x >> 10, ((x & ((1 << 10) - 1)) * 100) >> 10);\n \t} else {\n-\t\tl -= snprintf(tp->display, l, \", %u bytes\", (int)total);\n+\t\tl -= snprintf(tp->display, l, \"%s%u bytes\", delim, (int)total);\n \t}\n \n \tif (rate > 1 << 10) {\n@@ -197,7 +201,7 @@ void display_throughput(struct progress *progress, off_t total)\n \t\ttp->last_misecs[tp->idx] = misecs;\n \t\ttp->idx = (tp->idx + 1) % TP_IDX_MAX;\n \n-\t\tthroughput_string(tp, total, rate);\n+\t\tthroughput_string(tp, total, rate, progress);\n \t\tif (progress->last_value != -1 && progress_update)\n \t\t\tdisplay(progress, progress->last_value, NULL);\n \t}\n@@ -255,7 +259,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)\n \t\tif (tp) {\n \t\t\tunsigned int rate = !tp->avg_misecs ? 0 :\n \t\t\t\t\ttp->avg_bytes / tp->avg_misecs;\n-\t\t\tthroughput_string(tp, tp->curr_total, rate);\n+\t\t\tthroughput_string(tp, tp->curr_total, rate, progress);\n \t\t}\n \t\tprogress_update = 1;\n \t\tsprintf(bufp, \", %s.\\n\", msg);\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179258","messageId":"20111110075312.GL27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 12/14] remote-curl: show progress for bundle downloads","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:53:12Z","receivedAt":"2011-11-10T07:53:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Generally, the point of fetching from a bundle is that it's\nbig. Without a progress meter, git will appear to hang\nduring the long download.\n\nThis patch adds a throughput meter (i.e., just the bytes\ntransferred and the rate). In the long run, we should look\nfor a content-length header from the server so we can show a\ntotal size and completion percentage. However, displaying\nthat properly will require some surgery to the progress\ncode, so let's leave it as a future enhancement.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n remote-curl.c |   18 +++++++++++++++++-\n 1 files changed, 17 insertions(+), 1 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 7734495..6b0820e 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -8,6 +8,7 @@\n #include \"pkt-line.h\"\n #include \"sideband.h\"\n #include \"bundle.h\"\n+#include \"progress.h\"\n \n static struct remote *remote;\n static const char *url; /* always ends with a trailing slash */\n@@ -107,6 +108,9 @@ struct get_refs_cb_data {\n \tint is_bundle;\n \tconst char *tmpname;\n \tFILE *fh;\n+\n+\tstruct progress *progress;\n+\toff_t total;\n };\n \n static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)\n@@ -114,8 +118,11 @@ static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)\n \tstruct get_refs_cb_data *data = vdata;\n \tstruct strbuf *out = data->out;\n \n-\tif (data->is_bundle > 0)\n+\tif (data->is_bundle > 0) {\n+\t\tdata->total += sz * n;\n+\t\tdisplay_throughput(data->progress, data->total);\n \t\treturn fwrite(buf, sz, n, data->fh);\n+\t}\n \n \tstrbuf_add(out, buf, sz * n);\n \n@@ -129,6 +136,12 @@ static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)\n \t\t\tdie_errno(\"unable to open %s\", data->tmpname);\n \t\tif (fwrite(out->buf, 1, out->len, data->fh) < out->len)\n \t\t\tdie_errno(\"unable to write to %s\", data->tmpname);\n+\t\tif (options.progress) {\n+\t\t\tdata->total = out->len;\n+\t\t\tdata->progress = start_progress(\"Downloading bundle\", 0);\n+\t\t\tdisplay_progress(data->progress, 0);\n+\t\t\tdisplay_throughput(data->progress, data->total);\n+\t\t}\n \t}\n \treturn sz * n;\n }\n@@ -143,6 +156,8 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,\n \tdata.is_bundle = -1;\n \tdata.tmpname = tmpname;\n \tdata.fh = NULL;\n+\tdata.progress = NULL;\n+\tdata.total = 0;\n \n \tret = http_get_callback(url, get_refs_callback, &data, 0, options);\n \n@@ -150,6 +165,7 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,\n \t\tif (fclose(data.fh))\n \t\t\tdie_errno(\"unable to write to %s\", data.tmpname);\n \t}\n+\tstop_progress(&data.progress);\n \n \t*is_bundle = data.is_bundle > 0;\n \treturn ret;\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179259","messageId":"20111110075353.GM27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 13/14] remote-curl: resume interrupted bundle transfers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:53:53Z","receivedAt":"2011-11-10T07:53:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"If we have a bundle file from a previous fetch that matches\nthis URL, then we should resume the transfer where we left\noff.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nThe second half of the diff is hard to read because I re-indent a big\nchunk. It's much easier to see what's going on with \"diff -b\".\n\n remote-curl.c |   64 +++++++++++++++++++++++++++++++++++++++++++++++---------\n 1 files changed, 53 insertions(+), 11 deletions(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 6b0820e..43ad1b6 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -9,6 +9,7 @@\n #include \"sideband.h\"\n #include \"bundle.h\"\n #include \"progress.h\"\n+#include \"dir.h\"\n \n static struct remote *remote;\n static const char *url; /* always ends with a trailing slash */\n@@ -171,6 +172,32 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,\n \treturn ret;\n }\n \n+static int resume_bundle(const char *url, const char *tmpname)\n+{\n+\tstruct get_refs_cb_data data;\n+\tint ret;\n+\n+\tdata.fh = fopen(tmpname, \"ab\");\n+\tif (!data.fh)\n+\t\tdie_errno(\"unable to open %s\", tmpname);\n+\n+\tdata.is_bundle = 1;\n+\tdata.total = ftell(data.fh);\n+\tif (options.progress) {\n+\t\tdata.progress = start_progress(\"Resuming bundle\", 0);\n+\t\tdisplay_progress(data.progress, 0);\n+\t\tdisplay_throughput(data.progress, data.total);\n+\t}\n+\n+\tret = http_get_callback(url, get_refs_callback, &data, data.total, 0);\n+\n+\tif (fclose(data.fh))\n+\t\tdie_errno(\"unable to write to %s\", tmpname);\n+\tstop_progress(&data.progress);\n+\n+\treturn ret;\n+}\n+\n static const char *url_to_bundle_tmpfile(const char *url)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -210,20 +237,35 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,\n \tfree_discovery(last);\n \n \tfilename = url_to_bundle_tmpfile(url);\n+\tif (file_exists(filename)) {\n+\t\tstruct strbuf trimmed = STRBUF_INIT;\n \n-\tstrbuf_addf(&buffer, \"%sinfo/refs\", url);\n-\tif (!prefixcmp(url, \"http://\") || !prefixcmp(url, \"https://\")) {\n-\t\tis_http = 1;\n-\t\tif (!strchr(url, '?'))\n-\t\t\tstrbuf_addch(&buffer, '?');\n-\t\telse\n-\t\t\tstrbuf_addch(&buffer, '&');\n-\t\tstrbuf_addf(&buffer, \"service=%s\", service);\n+\t\tstrbuf_addstr(&trimmed, url);\n+\t\twhile (trimmed.len > 0 && trimmed.buf[trimmed.len-1] == '/')\n+\t\t\tstrbuf_setlen(&trimmed, trimmed.len - 1);\n+\t\trefs_url = strbuf_detach(&trimmed, NULL);\n+\n+\t\thttp_ret = resume_bundle(refs_url, filename);\n+\t\tis_bundle = 1;\n \t}\n-\trefs_url = strbuf_detach(&buffer, NULL);\n+\telse\n+\t\thttp_ret = HTTP_MISSING_TARGET;\n+\n+\tif (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {\n+\t\tstrbuf_addf(&buffer, \"%sinfo/refs\", url);\n+\t\tif (!prefixcmp(url, \"http://\") || !prefixcmp(url, \"https://\")) {\n+\t\t\tis_http = 1;\n+\t\t\tif (!strchr(url, '?'))\n+\t\t\t\tstrbuf_addch(&buffer, '?');\n+\t\t\telse\n+\t\t\t\tstrbuf_addch(&buffer, '&');\n+\t\t\tstrbuf_addf(&buffer, \"service=%s\", service);\n+\t\t}\n+\t\trefs_url = strbuf_detach(&buffer, NULL);\n \n-\thttp_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE,\n-\t\t\t\t     filename, &is_bundle);\n+\t\thttp_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE,\n+\t\t\t\t\t     filename, &is_bundle);\n+\t}\n \n \t/* try again with \"plain\" url (no ? or & appended) */\n \tif (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179260","messageId":"20111110075654.GN27950@sigill.intra.peff.net","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"[PATCH 14/14] clone: give advice on how to resume a failed clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-10T07:56:55Z","receivedAt":"2011-11-10T07:56:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"When clone fails, we usually delete the partial directory.\nHowever, if cloning was fetching a bundle, that is resumable\nand we should consider those results precious.\n\nThis patch detects when a partial bundle is present,\npreserves the directory, and gives the user some advice\nabout how to resume.\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\nWe could make \"git clone ...\" automatically resume, but I'm a little\nnervous about that. I wrote a patch that did so, and it did work, but\nthere are a lot of little hiccups as we violate the assumption that the\ndirectory didn't already exist (e.g., it writes multiple fetch refspec\nlines to the config).\n\nBut more importantly, I really worry about destroying the safety valve\nof not overwriting an existing directory. Yes, we can check to see that\nit is a git directory and that it has a partially-downloaded bundle\nfile. But that could also describe an existing git repo with unstaged\nchanges. And you sure don't want to \"checkout -f\" over them.\n\nSo I'd rather at least start with giving the user some advice and\nhaving them explicitly say \"yeah, I do want to resume this\".\n\n builtin/clone.c |   34 ++++++++++++++++++++++++++++++++++\n 1 files changed, 34 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex efe8b6c..c242e20 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -392,6 +392,36 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,\n \treturn ret;\n }\n \n+static int git_dir_is_resumable(const char *dir)\n+{\n+\tconst char *objects = mkpath(\"%s/objects\", dir);\n+\tDIR *dh = opendir(objects);\n+\tstruct dirent *de;\n+\n+\tif (!dh)\n+\t\treturn 0;\n+\n+\twhile ((de = readdir(dh))) {\n+\t\tif (!prefixcmp(de->d_name, \"tmp_bundle_\")) {\n+\t\t\tclosedir(dh);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\n+\tclosedir(dh);\n+\treturn 0;\n+}\n+\n+static void give_resume_advice(void)\n+{\n+\tadvise(\"Cloning failed, but partial results were saved.\");\n+\tadvise(\"You can resume the fetch with:\");\n+\tadvise(\"  git fetch\");\n+\tif (!option_bare)\n+\t\tadvise(\"  git checkout %s\",\n+\t\t       option_branch ? option_branch : \"master\");\n+}\n+\n static const char *junk_work_tree;\n static const char *junk_git_dir;\n static pid_t junk_pid;\n@@ -402,6 +432,10 @@ static void remove_junk(void)\n \tif (getpid() != junk_pid)\n \t\treturn;\n \tif (junk_git_dir) {\n+\t\tif (git_dir_is_resumable(junk_git_dir)) {\n+\t\t\tgive_resume_advice();\n+\t\t\treturn;\n+\t\t}\n \t\tstrbuf_addstr(&sb, junk_git_dir);\n \t\tremove_dir_recursively(&sb, 0);\n \t\tstrbuf_reset(&sb);\n-- \n1.7.7.2.7.g9f96f\n"},{"id":"179269","messageId":"alpine.DEB.2.00.1111100941380.27581@tvnag.unkk.fr","threadId":"28903","inReplyTo":"20111110074803.GB27950@sigill.intra.peff.net","subject":"Re: [PATCH 02/14] http: turn off curl signals","fromName":"Daniel Stenberg","fromEmail":"daniel@haxx.se","sentAt":"2011-11-10T08:43:40Z","receivedAt":"2011-11-10T08:43:40Z","isPatch":true,"sender":{"key":"daniel@haxx.se","avatar":"https://gravatar.com/avatar/69fdca87edd17cee21ca2e79fc2ff671d644603c3dc27167430f3cd3dbab7ba8?d=mp&s=160"},"body":"On Thu, 10 Nov 2011, Jeff King wrote:\n\n> I'm a little iffy on this one. If I understand correctly, depending on the \n> build and configuration, curl may not be able to timeout during DNS lookups. \n> But I'm not sure if it does, anyway, since we don't set any timeouts.\n\nRight, without a timeout set libcurl won't try to timeout name resolves.\n\nTo clarify: when libcurl is built to use the standard synchronous name \nresolver functions it can only abort them after a specified time by using \nsignals (on posix systems).\n\n-- \n\n  / daniel.haxx.se\n"},{"id":"179292","messageId":"7vvcqrsmfx.fsf@alter.siamese.dyndns.org","threadId":"28903","inReplyTo":"20111110075654.GN27950@sigill.intra.peff.net","subject":"Re: [PATCH 14/14] clone: give advice on how to resume a failed clone","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-11-10T21:21:38Z","receivedAt":"2011-11-10T21:21:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> We could make \"git clone ...\" automatically resume, but I'm a little\n> nervous about that. I wrote a patch that did so, and it did work, but\n> there are a lot of little hiccups as we violate the assumption that the\n> directory didn't already exist (e.g., it writes multiple fetch refspec\n> lines to the config).\n\nSorry, but I do not think the worry is quite justified.\n\nThe \"assumption that directory didn't already exist\" becomes an issue only\nif you implement your \"git clone\" that automatically resumes as a thin\nwrapper around the current \"git clone\" in the form of\n\n    until git clone ...\n    do\n\techo retrying...\n    done\n\nStepping back a bit, I think there are two different situations where\nresumable clone is beneficial. The \"git clone\" process died either by the\nmachine crashing or the user hitting a \\C-c, or the connection between the\nserver and the \"git clone\" got severed for some reason.\n\nRight now, the \"got disconnected\" case results in \"git clone\" voluntarily\ndying and as the result of this, the symptom appears the same.  But it\ndoes not have to be that way.  If we know the underlying transport is\nresumable, e.g. when you are fetching a prepared bundle over the wire.\n\nI have this suspicion that in practice the \"got disconnected\" case is the\nmajority. If \"git clone\" does not die upon disconnect while fetching a\nbundle, but instead the fetching of the bundle is resumed internally by\nreconnecting to the server and requesting a range transfer, there is no\nrisk of \"writes multiple fetch refspec lines\" etc, no?\n\nOf course, it is _also_ beneficial if we made \"git clone\" resumable after\nyou purposefully kill it (maybe you thought it will clone within minutes,\nbut it turns out that it may take hours and you have to turn off the\nmachine in the next five minutes before leaving the work, or something).\nA solution for that case _could_ be used for the \"got disconnected\" case\nby letting it voluntarily die as we currently do, but I do not think that\nis an optimal solution to the \"got disconnected\" case.\n"},{"id":"179314","messageId":"CAFfmPPN=f2nL4NAbhnHhkKvMFpgi2B+DeUQ42Si+do=Z4_SAXQ@mail.gmail.com","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"Re: [PATCH 0/14] resumable network bundles","fromName":"David Michael Barr","fromEmail":"davidbarr@google.com","sentAt":"2011-11-11T13:13:07Z","receivedAt":"2011-11-11T13:13:07Z","isPatch":true,"sender":{"key":"davidbarr@google.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"On Thu, Nov 10, 2011 at 6:43 PM, Jeff King <peff@peff.net> wrote:\n> One possible option for resumable clones that has been discussed is\n> letting the server point the client by http to a static bundle\n> containing most of history, followed by a fetch from the actual git repo\n> (which should be much cheaper now that we have all of the bundled\n> history).  This series implements \"step 0\" of this plan: just letting\n> bundles be fetched across the network in the first place.\n>\n> Shawn raised some issues about using bundles for this (as opposed to\n> accessing the packfiles themselves); specifically, this raises the I/O\n> footprint of a repository that has to serve both the bundled version of\n> the pack and the regular packfile.\n>\n> So it may be that we don't follow this plan all the way through.\n> However, even if we don't, fetching bundles over http is still a useful\n> thing to be able to do. Which makes this first step worth doing either\n> way.\n>\n>  [01/14]: t/lib-httpd: check for NO_CURL\n>  [02/14]: http: turn off curl signals\n>  [03/14]: http: refactor http_request function\n>  [04/14]: http: add a public function for arbitrary-callback request\n>  [05/14]: remote-curl: use http callback for requesting refs\n>  [06/14]: transport: factor out bundle to ref list conversion\n>  [07/14]: bundle: add is_bundle_buf helper\n>  [08/14]: remote-curl: free \"discovery\" object\n>  [09/14]: remote-curl: auto-detect bundles when fetching refs\n>  [10/14]: remote-curl: try base $URL after $URL/info/refs\n>  [11/14]: progress: allow pure-throughput progress meters\n>  [12/14]: remote-curl: show progress for bundle downloads\n>  [13/14]: remote-curl: resume interrupted bundle transfers\n>  [14/14]: clone: give advice on how to resume a failed clone\n>\n> -Peff\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n\nI just want to say thank you for doing this.--David Barr\n"},{"id":"179327","messageId":"20111111205256.GB20515@sigill.intra.peff.net","threadId":"28903","inReplyTo":"7vvcqrsmfx.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 14/14] clone: give advice on how to resume a failed clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-11T20:52:56Z","receivedAt":"2011-11-11T20:52:56Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 10, 2011 at 01:21:38PM -0800, Junio C Hamano wrote:\n\n> Jeff King <peff@peff.net> writes:\n> \n> > We could make \"git clone ...\" automatically resume, but I'm a little\n> > nervous about that. I wrote a patch that did so, and it did work, but\n> > there are a lot of little hiccups as we violate the assumption that the\n> > directory didn't already exist (e.g., it writes multiple fetch refspec\n> > lines to the config).\n> \n> Sorry, but I do not think the worry is quite justified.\n> \n> The \"assumption that directory didn't already exist\" becomes an issue only\n> if you implement your \"git clone\" that automatically resumes as a thin\n> wrapper around the current \"git clone\" in the form of\n> \n>     until git clone ...\n>     do\n> \techo retrying...\n>     done\n\nThat was sort of what my patch looked like. It didn't do the wrapper\nbit; you would have to run \"git clone\" again to resume. I.e.:\n\n  $ git clone http://...\n  Downloading bundle: ...\n  ^C\n  $ git clone http://...\n  Resuming bundle: ...\n\nand the implementation was very minimal. Basically, instead of checking\nthat the directory exists and dying, it said \"the directory exists, but\nthere is a resumable bundle in it, so let's keep going anyway\".\n\nYou _could_ do that right, with clone saying \"I got to the fetch stage\",\nand picking up there. But that means picking apart clone into its\nvarious stages, and not repeating earlier stages (or making sure they're\nidempotent).  And making sure that internally, later stages don't depend\non in-memory actions from the earlier stages.\n\nAnd then what happens when you have different parameters? If I say \"git\nclone foo\", and then resume with \"git clone --mirror foo\", what should\nhappen?\n\nNone of these problems are insurmountable. But it means looking over the\nclone code carefully, figuring out what should happen, and then probably\nbreaking apart the various stages to see where we can resume from.\n\nI wanted to start simply, and just tell the user \"this is approximately\nwhat clone would have done from here\". And then the fancier automatic\nbits can come later.\n\n> Stepping back a bit, I think there are two different situations where\n> resumable clone is beneficial. The \"git clone\" process died either by the\n> machine crashing or the user hitting a \\C-c, or the connection between the\n> server and the \"git clone\" got severed for some reason.\n> \n> Right now, the \"got disconnected\" case results in \"git clone\" voluntarily\n> dying and as the result of this, the symptom appears the same.  But it\n> does not have to be that way.  If we know the underlying transport is\n> resumable, e.g. when you are fetching a prepared bundle over the wire.\n>\n> I have this suspicion that in practice the \"got disconnected\" case is the\n> majority. If \"git clone\" does not die upon disconnect while fetching a\n> bundle, but instead the fetching of the bundle is resumed internally by\n> reconnecting to the server and requesting a range transfer, there is no\n> risk of \"writes multiple fetch refspec lines\" etc, no?\n\nI don't think it is the majority. And there are even variants of \"git\ndisconnected\" that don't work. Here are just a few cases it wouldn't\nhandle that I think are common:\n\n  1. the client machine crashes or loses power; you'd like to resume\n     after rebooting.\n\n  2. the network or server goes out, but is not immediately available.\n     The followup attempt to fetch fails, but you could succeed if you\n     restarted the fetch a few minutes (or hours, or days) later.\n\n  3. the user hits ^C not because they want to abort the clone entirely,\n     but because they know they cannot complete the clone right now\n     (e.g., they are taking their laptop off the network, or it is\n     consuming too much bandwidth, or they would rather wait until later\n     when they are on a faster network).\n\nAll of those mean we have to have some on-disk state that shows how far\nwe got, and that a totally separate process can figure out from the\nstate where to resume.\n\n> Of course, it is _also_ beneficial if we made \"git clone\" resumable after\n> you purposefully kill it (maybe you thought it will clone within minutes,\n> but it turns out that it may take hours and you have to turn off the\n> machine in the next five minutes before leaving the work, or something).\n> A solution for that case _could_ be used for the \"got disconnected\" case\n> by letting it voluntarily die as we currently do, but I do not think that\n> is an optimal solution to the \"got disconnected\" case.\n\nI see the \"got disconnected, and we can immediately resume\" as a\nminority subset of the larger problem. If we really want to, we can\nimplement that while waiting for a larger solution, but I don't think it\nwill serve most people's needs, and then will eventually become obsolete\nanyway.\n\n-Peff\n"},{"id":"179328","messageId":"20111111205442.GC20515@sigill.intra.peff.net","threadId":"28903","inReplyTo":"alpine.DEB.2.00.1111100941380.27581@tvnag.unkk.fr","subject":"Re: [PATCH 02/14] http: turn off curl signals","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-11T20:54:42Z","receivedAt":"2011-11-11T20:54:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 10, 2011 at 09:43:40AM +0100, Daniel Stenberg wrote:\n\n> >I'm a little iffy on this one. If I understand correctly, depending\n> >on the build and configuration, curl may not be able to timeout\n> >during DNS lookups. But I'm not sure if it does, anyway, since we\n> >don't set any timeouts.\n> \n> Right, without a timeout set libcurl won't try to timeout name resolves.\n> \n> To clarify: when libcurl is built to use the standard synchronous\n> name resolver functions it can only abort them after a specified time\n> by using signals (on posix systems).\n\nOK, that matches with my understanding. I think this patch is a fine\nthing to do for us, then. If we ever do start caring about timing out on\nname lookups, we can rework it.\n\nThanks.\n\n-Peff\n"},{"id":"179352","messageId":"CALUzUxqsLP11Tcsoc1tzGMfNcMqor2wQF+Yutu3FRiTin4Pnew@mail.gmail.com","threadId":"28903","inReplyTo":"20111110074330.GA27925@sigill.intra.peff.net","subject":"Re: [PATCH 0/14] resumable network bundles","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2011-11-12T16:11:31Z","receivedAt":"2011-11-12T16:11:31Z","isPatch":true,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"On Thu, Nov 10, 2011 at 3:43 PM, Jeff King <peff@peff.net> wrote:\n> One possible option for resumable clones that has been discussed is\n> letting the server point the client by http to a static bundle\n> containing most of history, followed by a fetch from the actual git repo\n> (which should be much cheaper now that we have all of the bundled\n> history).  This series implements \"step 0\" of this plan: just letting\n> bundles be fetched across the network in the first place.\n>\n> Shawn raised some issues about using bundles for this (as opposed to\n> accessing the packfiles themselves); specifically, this raises the I/O\n> footprint of a repository that has to serve both the bundled version of\n> the pack and the regular packfile.\n>\n> So it may be that we don't follow this plan all the way through.\n> However, even if we don't, fetching bundles over http is still a useful\n> thing to be able to do. Which makes this first step worth doing either\n> way.\n\nJeff, this is a great series, I think the cleanups and refactors\nshould get integrated independently of the bundle-cloning stuff.\n\nOne thing I'm not comfortable with is the \"flexibility\" allowed in\nbundle fetching - servers are allowed to send bundles if they see fit,\nand we have to detect it when they do (if I'm reading the \"surprised\"\nscenario in patch 9 correctly).\n\nPerhaps we can expose bundle fetching through /objects/info/bundles?\nIt could possibly contain information about what bundles are available\nand what revs they contain. If bundles are found, fetch them;\notherwise, go through the usual ref advertisement and other steps of\nthe pack protocol.\n\nThat way, we take out the \"surprise\" factor in the fetching protocol.\n\n-- \nCheers,\nRay Chuan\n"},{"id":"179356","messageId":"20111112175826.GA9427@sigill.intra.peff.net","threadId":"28903","inReplyTo":"CALUzUxqsLP11Tcsoc1tzGMfNcMqor2wQF+Yutu3FRiTin4Pnew@mail.gmail.com","subject":"Re: [PATCH 0/14] resumable network bundles","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-11-12T17:58:26Z","receivedAt":"2011-11-12T17:58:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Nov 13, 2011 at 12:11:31AM +0800, Tay Ray Chuan wrote:\n\n> One thing I'm not comfortable with is the \"flexibility\" allowed in\n> bundle fetching - servers are allowed to send bundles if they see fit,\n> and we have to detect it when they do (if I'm reading the \"surprised\"\n> scenario in patch 9 correctly).\n\nRight.\n\n> Perhaps we can expose bundle fetching through /objects/info/bundles?\n\nBut what if the server you are hitting doesn't have a git repo at all?\nIn the simplest case, a bundle provider should just be able to put a\nfile somewhere http-ccessible, without having any special directory\nstructure or other meta files.\n\nWhich means that we have to be prepared for the URL the user gave us to\nbe a bundle, not a git repo that contains bundles.\n\n> It could possibly contain information about what bundles are available\n> and what revs they contain. If bundles are found, fetch them;\n> otherwise, go through the usual ref advertisement and other steps of\n> the pack protocol.\n\nThis is \"step 2\" of my plan: hitting a git repo will provide a way of\nredirecting to other, static storage. But I think it's important that\nthe other storage not just be a path in the existing repo, for two\nreasons:\n\n  1. You might want to redirect the client off-server to a\n     higher-bandwidth static service like S3, or something backed by a\n     CDN.\n\n  2. The client might not be hitting you through http, so you can't\n     expect them to look at arbitrary repo files (like\n     objects/info/bundles). We need to provide the information over the\n     git protocol (my plan is to use a special ref name, like\n     \"refs/mirrors\" to encode the information).\n\n> That way, we take out the \"surprise\" factor in the fetching protocol.\n\nI don't think it's that big a deal. It influenced the way that patches 9\nand 10 were written (patch 9 handles \"surprise\" bundles when fetching\ninfo/refs, and then patch 10 falls back to fetching $URL without\ninfo/refs). But even if we didn't have the \"surprise\" case, most of the\ncode in patch 9 would have just ended up in patch 10. That is, the\nsurprise case doesn't take much code, and doesn't have a negative impact\non the non-surprise case (i.e., until we see a bundle header, the\nbehavior is identical, just putting the refs into a memory buffer).\n\n-Peff\n"}]}