{"thread":{"id":"21746","subject":"Client-side mirroring patches (v0)","startedAt":"2009-11-25T10:06:53Z","lastAt":"2010-01-01T00:05:46Z","messageCount":8,"participants":["Sam Vilain","Shawn O. Pearce","Nanako Shiraishi"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"128295","messageId":"1259143617-26580-1-git-send-email-sam@vilain.net","threadId":"21746","inReplyTo":null,"subject":"Client-side mirroring patches (v0)","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-11-25T10:06:53Z","receivedAt":"2009-11-25T10:06:53Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Hey folks, this is the first stage of git mirroring - making the\nclient support it in the face of a completely ignorant server.  I\nintended to make the clean-up better, but I've been sitting on these\nfor a couple of weeks so I thought it would be better to have them out\nthere for people to have a squiz at.\n\nThe next stage would be for the mirror list to be communicated to\nclients over the network protocol and updated in the git config.\n\nAlso there is the matter of falling over to the next mirror should one\nnot be reachable, but then we're getting into C weaknesses really.\nShould I plan to do exception recovery using 'longjmp' ?  Also the\nprocess should be interruptible and provide a user menu.  Again this\nseems like it would be very tedious and clumsy in C.  How do people\nmanage?  Anyway, enjoy... Sam\n"},{"id":"128298","messageId":"1259143617-26580-2-git-send-email-sam@vilain.net","threadId":"21746","inReplyTo":"1259143617-26580-1-git-send-email-sam@vilain.net","subject":"[PATCH 1/4] remote: allow mirroring to be specified, and document settings","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-11-25T10:06:54Z","receivedAt":"2009-11-25T10:06:54Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Add a per-remote setting that can list alternate URLs that the same\nrepository (or a close fork) can be found, a fetch command-line\nswitch, a config flag to enable it by default, and finally a method\nfor selecting a default mirror.\n\nIf the preferred mirror does not exist in the list of mirrors it is\nconsidered invalid, in preparation for the time when the mirror list\ncan be supplied or added to by the upload-pack protocol response.\n\nSigned-off-by: Sam Vilain <sam@vilain.net>\n---\n Documentation/config.txt        |   13 +++++++++++++\n Documentation/fetch-options.txt |   24 ++++++++++++++++++++++++\n builtin-fetch.c                 |    4 +++-\n remote.c                        |   17 +++++++++++++++++\n remote.h                        |    6 ++++++\n 5 files changed, 63 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex d1e2120..edde0e4 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1375,6 +1375,19 @@ remote.<name>.url::\n remote.<name>.pushurl::\n \tThe push URL of a remote repository.  See linkgit:git-push[1].\n \n+remote.<name>.mirror-url::\n+\tAn alternate URL which should have many similar refs to the\n+\treal remote at least most of the time.  This option can be\n+\tspecified multiple times.  See linkgit:git-fetch[1].\n+\n+remote.<name>.use-mirror::\n+\tPrefer to contact mirrors first (boolean).  See\n+\tlinkgit:git-fetch[1].\n+\n+remote.<name>.preferred-mirror::\n+\tSpecify which mirror to try first (full URL).  May be updated\n+\tby user interaction during 'fetch'.  See linkgit:git-fetch[1].\n+\n remote.<name>.proxy::\n \tFor remotes that require curl (http, https and ftp), the URL to\n \tthe proxy to use for that remote.  Set to the empty string to\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 2886874..8f07a56 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -62,6 +62,30 @@ ifndef::git-pull[]\n \tPass --quiet to git-fetch-pack and silence any other internally\n \tused git commands.\n \n+-M::\n+--use-mirror::\n+\tTry to use a configured mirror for the bulk of the transfer\n+\trather than the main upstream.  See gitlink:git-config[1] for\n+\tthe appropriate options to set.\n++\n+The preferred mirror (or the first if no preferred URL is defined) is\n+first contacted and fetched from, and any refs it presents which match\n+the source <refspec> (including the rules for fetching tags) are saved\n+under `refs/mirrors/`<remote>`/hostname`.  If there is a network\n+timeout, or the user interrupts the fetch process, the next mirror\n+will be tried.\n++\n+Once the fetch from the mirror is complete, the central host is\n+contacted and fetched from.  If the mirror was correct and up to date,\n+then no more data will be required from the central host.  At this\n+point, all of the refs under `refs/mirrors/`<remote> which are\n+reachable from the real `refs/remotes/`<remote> tracking branches will\n+be removed.  Extra refs which were not present on the real source will\n+be left behind so they are not fetched again the next time around.\n++\n+This can be made the default for a remote using the\n+`remote.`<remote>.`use-mirror` configuration option.\n+\n -v::\n --verbose::\n \tBe verbose.\ndiff --git a/builtin-fetch.c b/builtin-fetch.c\nindex a35a6f8..209f502 100644\n--- a/builtin-fetch.c\n+++ b/builtin-fetch.c\n@@ -23,7 +23,7 @@ enum {\n \tTAGS_SET = 2\n };\n \n-static int append, force, keep, update_head_ok, verbosity;\n+static int append, force, keep, update_head_ok, verbosity, use_mirror;\n static int tags = TAGS_DEFAULT;\n static const char *depth;\n static const char *upload_pack;\n@@ -45,6 +45,8 @@ static struct option builtin_fetch_options[] = {\n \tOPT_BOOLEAN('k', \"keep\", &keep, \"keep downloaded pack\"),\n \tOPT_BOOLEAN('u', \"update-head-ok\", &update_head_ok,\n \t\t    \"allow updating of HEAD ref\"),\n+\tOPT_BOOLEAN('M', \"mirror\", &use_mirror,\n+\t\t    \"use mirror if available\"),\n \tOPT_STRING(0, \"depth\", &depth, \"DEPTH\",\n \t\t   \"deepen history of shallow clone\"),\n \tOPT_END()\ndiff --git a/remote.c b/remote.c\nindex 73d33f2..65df03d 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -111,6 +111,13 @@ static void add_pushurl(struct remote *remote, const char *pushurl)\n \tremote->pushurl[remote->pushurl_nr++] = pushurl;\n }\n \n+static void add_mirror_url(struct remote *remote, const char *mirror_url)\n+{\n+\tALLOC_GROW(remote->mirror_url, remote->mirror_url_nr + 1,\n+\t\t   remote->mirror_url_alloc);\n+\tremote->mirror_url[remote->mirror_url_nr++] = mirror_url;\n+}\n+\n static void add_pushurl_alias(struct remote *remote, const char *url)\n {\n \tconst char *pushurl = alias_url(url, &rewrites_push);\n@@ -407,6 +414,16 @@ static int handle_config(const char *key, const char *value, void *cb)\n \t\tif (git_config_string(&v, key, value))\n \t\t\treturn -1;\n \t\tadd_pushurl(remote, v);\n+\t} else if (!strcmp(subkey, \".mirror-url\")) {\n+\t\tconst char *v;\n+\t\tif (git_config_string(&v, key, value))\n+\t\t\treturn -1;\n+\t\tadd_mirror_url(remote, v);\n+\t} else if (!strcmp(subkey, \".use-mirror\")) {\n+\t\tremote->use_mirror = git_config_bool(key, value);\n+\t} else if (!strcmp(subkey, \".preferred-mirror\")) {\n+\t\tif (git_config_string(&remote->preferred_mirror, key, value))\n+\t\t\treturn -1;\n \t} else if (!strcmp(subkey, \".push\")) {\n \t\tconst char *v;\n \t\tif (git_config_string(&v, key, value))\ndiff --git a/remote.h b/remote.h\nindex 5db8420..c720b9a 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -19,6 +19,12 @@ struct remote {\n \tint pushurl_nr;\n \tint pushurl_alloc;\n \n+\tconst char **mirror_url;\n+\tint mirror_url_nr;\n+\tint mirror_url_alloc;\n+\tint use_mirror;\n+\tconst char *preferred_mirror;\n+\n \tconst char **push_refspec;\n \tstruct refspec *push;\n \tint push_refspec_nr;\n-- \n1.6.3.3\n"},{"id":"128299","messageId":"1259143617-26580-3-git-send-email-sam@vilain.net","threadId":"21746","inReplyTo":"1259143617-26580-2-git-send-email-sam@vilain.net","subject":"[PATCH 2/4] fetch: try mirrors if selected","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-11-25T10:06:55Z","receivedAt":"2009-11-25T10:06:55Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"If configured and selected, the mirrors are tried in turn until one\nsucceeds, re-writing the refs to a refs/mirrors/<remote>/<hostname>/ space.\nNo refs from the mirrors are ever written to the real refs/heads or\nrefs/tags spaces, but their being available locally will speed up fetching\nfrom the real remote if they are more up to date than the local version.\n\nSigned-off-by: Sam Vilain <sam@vilain.net>\n---\n builtin-fetch.c         |  161 ++++++++++++++++++++++++++++++++++++++++++++--\n remote.c                |   14 ++++-\n remote.h                |    1 +\n t/t5560-mirror-fetch.sh |   46 +++++++++++++\n transport.c             |   41 ++++++++++++\n transport.h             |    5 ++\n 6 files changed, 260 insertions(+), 8 deletions(-)\n create mode 100644 t/t5560-mirror-fetch.sh\n\ndiff --git a/builtin-fetch.c b/builtin-fetch.c\nindex 209f502..b3b8766 100644\n--- a/builtin-fetch.c\n+++ b/builtin-fetch.c\n@@ -45,8 +45,8 @@ static struct option builtin_fetch_options[] = {\n \tOPT_BOOLEAN('k', \"keep\", &keep, \"keep downloaded pack\"),\n \tOPT_BOOLEAN('u', \"update-head-ok\", &update_head_ok,\n \t\t    \"allow updating of HEAD ref\"),\n-\tOPT_BOOLEAN('M', \"mirror\", &use_mirror,\n-\t\t    \"use mirror if available\"),\n+\tOPT_SET_INT('M', \"use-mirror\", &use_mirror,\n+\t\t    \"use mirror if available\", 1),\n \tOPT_STRING(0, \"depth\", &depth, \"DEPTH\",\n \t\t   \"deepen history of shallow clone\"),\n \tOPT_END()\n@@ -109,6 +109,109 @@ static void find_non_local_tags(struct transport *transport,\n \t\t\tstruct ref **head,\n \t\t\tstruct ref ***tail);\n \n+char* get_url_hostname(const char *url)\n+{\n+\tchar *scratch = xstrdup(url);\n+\tchar *host = strstr(url, \"://\");\n+\tchar c;\n+\tchar *end, *rh;\n+\tif (host) {\n+\t\thost += 3;\n+\t\tc = '/';\n+\t}\n+\telse {\n+\t\thost = scratch;\n+\t\tc = ':';\n+\t}\n+\n+\tif (host[0] == '[') {\n+\t\tend = strchr(host + 1, ']');\n+\t\tif (end) {\n+\t\t\t*end = 0;\n+\t\t\thost++;\n+\t\t}\n+\t}\n+\telse {\n+\t\tend = strchr(host, c);\n+\t\tif (end && !has_dos_drive_prefix(url) ) {\n+\t\t\t*end = 0;\n+\t\t}\n+\t\telse {\n+\t\t\thost = \"localhost\";\n+\t\t}\n+\t}\n+\trh = xstrdup(host);\n+\tfree(scratch);\n+\treturn rh;\n+}\n+\n+const char *mirror_ref(const char* remote_name, const char* mirror_hostname,\n+\t\t       const char* refname)\n+{\n+\tint has_refs, new_sz;\n+\tchar *rv, *dst;\n+\n+\t// *rs[i] = *refspec[i];  ?\n+\thas_refs = ( strstr(refname, \"refs/\") == refname );\n+\t/* \"refs/\"(0 or 5) \"mirrors/\"(8) remote \"/\"(1) hostname \"/\"(1) */\n+\tnew_sz = (has_refs ? 0 : 5) + 8\n+\t\t+ strlen(remote_name) + 1\n+\t\t+ strlen(mirror_hostname) + 1\n+\t\t+ strlen(refname) + 1;\n+\trv = xmalloc( new_sz );\n+\tstrcpy(rv, \"refs/mirrors/\");\n+\tdst = rv + 13;\n+\tstrcpy(dst, remote_name);\n+\tdst += strlen(remote_name);\n+\t*dst++ = '/';\n+\tstrcpy(dst, mirror_hostname);\n+\tdst += strlen(mirror_hostname);\n+\t*dst++ = '/';\n+\tstrcpy(dst, refname+(has_refs?5:0));\n+\treturn rv;\n+}\n+\n+struct ref *mirror_refmap(struct transport* transport,\n+\t\t\t  struct ref* ref_map)\n+{\n+\tstruct ref *rm, *mirror_refmap, *last, *rv, *peer_ref;\n+\n+\tconst char* remote_name = transport->remote->name;\n+\tconst char* mirror_hostname = get_url_hostname(transport->url);\n+\tint c = 0;\n+\n+\tlast = NULL;\n+\trv = NULL;\n+\tfor (rm = ref_map; rm; rm = rm->next) {\n+\t\tconst char *new_dst;\n+\n+\t\t// skip refs we already have locally, to avoid ref churn\n+\t\tif (has_sha1_file(rm->old_sha1))\n+\t\t\tcontinue;\n+\n+\t\tmirror_refmap = alloc_ref(rm->name);\n+\t\tmirror_refmap->remote_status = rm->remote_status;\n+\t\thashcpy(mirror_refmap->old_sha1, rm->old_sha1);\n+\t\thashcpy(mirror_refmap->new_sha1, rm->new_sha1);\n+\n+\t\tif (last)\n+\t\t\tlast->next = mirror_refmap;\n+\t\telse\n+\t\t\trv = mirror_refmap;\n+\t\tc++;\n+\n+\t\tnew_dst = mirror_ref(remote_name, mirror_hostname, rm->name);\n+\n+\t\tpeer_ref = alloc_ref(new_dst);\n+\t\tmirror_refmap->peer_ref = peer_ref;\n+\t\tpeer_ref->force = 1;\n+\t\tlast = mirror_refmap;\n+\t}\n+\n+\treturn rv;\n+}\n+\n+\n static struct ref *get_ref_map(struct transport *transport,\n \t\t\t       struct refspec *refs, int ref_count, int tags,\n \t\t\t       int *autotags)\n@@ -165,10 +268,14 @@ static struct ref *get_ref_map(struct transport *transport,\n \tif (tags == TAGS_DEFAULT && *autotags)\n \t\tfind_non_local_tags(transport, &ref_map, &tail);\n \tref_remove_duplicates(ref_map);\n+\tif (strcmp(transport->url, transport->remote->url[0]) != 0) {\n+\t\treturn mirror_refmap(transport, ref_map);\n+\t}\n \n \treturn ref_map;\n }\n \n+\n #define STORE_REF_ERROR_OTHER 1\n #define STORE_REF_ERROR_DF_CONFLICT 2\n \n@@ -638,6 +745,7 @@ static int do_fetch(struct transport *transport,\n \t}\n \n \tref_map = get_ref_map(transport, refs, ref_count, tags, &autotags);\n+\n \tif (!update_head_ok)\n \t\tcheck_not_current_branch(ref_map);\n \n@@ -688,13 +796,18 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tint i;\n \tstatic const char **refs = NULL;\n \tint ref_nr = 0;\n-\tint exit_code;\n+\tint exit_code = 0;\n+\tint urls_remaining = 1;\n+\tstruct transport *real_transport = NULL;\n+\tconst char *mirror = NULL;\n+\tstruct refspec *refspec;\n \n \t/* Record the command line for the reflog */\n \tstrbuf_addstr(&default_rla, \"fetch\");\n \tfor (i = 1; i < argc; i++)\n \t\tstrbuf_addf(&default_rla, \" %s\", argv[i]);\n \n+\tuse_mirror = -1;\n \targc = parse_options(argc, argv, prefix,\n \t\t\t     builtin_fetch_options, builtin_fetch_usage, 0);\n \n@@ -706,6 +819,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (!remote)\n \t\tdie(\"Where do you want to fetch from today?\");\n \n+\tif (use_mirror == -1) {\n+\t\tuse_mirror = remote->use_mirror ? 1 : 0;\n+\t}\n+\n \ttransport = transport_get(remote, remote->url[0]);\n \tif (verbosity >= 2)\n \t\ttransport->verbose = 1;\n@@ -742,9 +859,39 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \n \tsigchain_push_common(unlock_pack_on_signal);\n \tatexit(unlock_pack);\n-\texit_code = do_fetch(transport,\n-\t\t\tparse_fetch_refspec(ref_nr, refs), ref_nr);\n-\ttransport_disconnect(transport);\n-\ttransport = NULL;\n+\trefspec = parse_fetch_refspec(ref_nr, refs);\n+\tif (use_mirror) {\n+\t\treal_transport = transport;\n+\t\turls_remaining = remote->mirror_url_nr + 1;\n+\t}\n+\twhile (urls_remaining) {\n+\t\tif (use_mirror && (urls_remaining > 1) ) {\n+\t\t\ttransport = transport_next_mirror(real_transport, mirror);\n+\t\t\tmirror = transport->url;\n+\t\t\twarning(\"trying mirror: %s\", mirror);\n+\t\t\t// real_transport may not have these options - re-set them.\n+\t\t\tif (upload_pack)\n+\t\t\t\tset_option(TRANS_OPT_UPLOADPACK, upload_pack);\n+\t\t\tif (keep)\n+\t\t\t\tset_option(TRANS_OPT_KEEP, \"yes\");\n+\t\t\tif (depth)\n+\t\t\t\tset_option(TRANS_OPT_DEPTH, depth);\n+\n+\t\t}\n+\t\texit_code = do_fetch(transport, refspec, ref_nr);\n+\t\ttransport_disconnect(transport);\n+\t\ttransport = NULL;\n+\t\turls_remaining--;\n+\t\tif (use_mirror) {\n+\t\t\tif (!exit_code && urls_remaining >= 1) {\n+\t\t\t\twarning(\"successful fetch from mirror\");\n+\t\t\t\turls_remaining = 1;\n+\t\t\t}\n+\t\t\tif (urls_remaining == 1) {\n+\t\t\t\ttransport = real_transport;\n+\t\t\t\twarning(\"trying master: %s\", transport->url);\n+\t\t\t}\n+\t\t}\n+\t}\n \treturn exit_code;\n }\ndiff --git a/remote.c b/remote.c\nindex 65df03d..5f08e10 100644\n--- a/remote.c\n+++ b/remote.c\n@@ -139,8 +139,9 @@ static struct remote *make_remote(const char *name, int len)\n \tfor (i = 0; i < remotes_nr; i++) {\n \t\tif (len ? (!strncmp(name, remotes[i]->name, len) &&\n \t\t\t   !remotes[i]->name[len]) :\n-\t\t    !strcmp(name, remotes[i]->name))\n+\t\t    !strcmp(name, remotes[i]->name)) {\n \t\t\treturn remotes[i];\n+\t\t}\n \t}\n \n \tret = xcalloc(1, sizeof(struct remote));\n@@ -683,6 +684,7 @@ static struct refspec *parse_push_refspec(int nr_refspec, const char **refspec)\n \treturn parse_refspec_internal(nr_refspec, refspec, 0, 0);\n }\n \n+\n static int valid_remote_nick(const char *name)\n {\n \tif (!name[0] || is_dot_or_dotdot(name))\n@@ -786,6 +788,16 @@ int remote_has_url(struct remote *remote, const char *url)\n \treturn 0;\n }\n \n+int remote_mirror_idx(struct remote *remote, const char *mirror_url)\n+{\n+\tint i;\n+\tfor (i = 0; i < remote->mirror_url_nr; i++) {\n+\t\tif (!strcmp(remote->mirror_url[i], mirror_url))\n+\t\t\treturn i;\n+\t}\n+\treturn -1;\n+}\n+\n static int match_name_with_pattern(const char *key, const char *name,\n \t\t\t\t   const char *value, char **result)\n {\ndiff --git a/remote.h b/remote.h\nindex c720b9a..da208ff 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -61,6 +61,7 @@ typedef int each_remote_fn(struct remote *remote, void *priv);\n int for_each_remote(each_remote_fn fn, void *priv);\n \n int remote_has_url(struct remote *remote, const char *url);\n+int remote_mirror_idx(struct remote *remote, const char *mirror_url);\n \n struct refspec {\n \tunsigned force : 1;\ndiff --git a/t/t5560-mirror-fetch.sh b/t/t5560-mirror-fetch.sh\nnew file mode 100644\nindex 0000000..940dc0e\n--- /dev/null\n+++ b/t/t5560-mirror-fetch.sh\n@@ -0,0 +1,46 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2009 Sam Vilain\n+#\n+\n+test_description='mirror fetch test'\n+\n+. ./test-lib.sh\n+\n+test_expect_success setup '\n+\techo >file master initial &&\n+\tgit add file &&\n+\tgit commit -a -m \"Master initial\" &&\n+\tgit clone . master &&\n+\tgit clone master mirror &&\n+\tcd master &&\n+\techo >file master update &&\n+\tgit commit -a -m \"Master update\" &&\n+\tcd .. &&\n+\tmkdir clone &&\n+\tcd clone &&\n+\tgit init &&\n+\tgit remote add origin ../master &&\n+\tgit config remote.origin.mirror-url ../mirror\n+'\n+\n+# in later iterations we'll expect these mirror tracking refs to be\n+# cleaned up once they are confirmed reachable from the master, but\n+# for now they leave a sufficient breadcrumb of the operation\n+\n+test_expect_success 'fetch using mirror - explicit' '\n+\tgit fetch --use-mirror origin refs/heads/*:refs/remotes/origin/* &&\n+\tgit rev-parse refs/mirrors/origin/localhost/heads/master\n+'\n+\n+test_expect_success 'fetch using mirror - default' '\n+\tcd .. &&\n+\tmkdir clone2 &&\n+\tcd clone2 &&\n+\tgit init &&\n+\tgit remote add origin ../master &&\n+\tgit config remote.origin.mirror-url ../mirror\n+\tgit fetch --use-mirror &&\n+\tgit rev-parse refs/mirrors/origin/localhost/heads/master\n+'\n+test_done\ndiff --git a/transport.c b/transport.c\nindex 644a30a..0dc0185 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -859,6 +859,47 @@ struct transport *transport_get(struct remote *remote, const char *url)\n \treturn ret;\n }\n \n+struct transport *transport_next_mirror(struct transport *transport,\n+\t\t\t\t\tconst char *last_mirror)\n+{\n+\tstruct transport *ret;\n+\tstruct remote* remote = transport->remote;\n+\tint mirror_idx = -1;\n+\tconst char* url;\n+\n+\tif (!last_mirror) {\n+\t\tif (remote->preferred_mirror) {\n+\t\t\tmirror_idx = remote_mirror_idx(\n+\t\t\t\tremote,\n+\t\t\t\tremote->preferred_mirror\n+\t\t\t\t);\n+\t\t\tif (mirror_idx == -1) {\n+\t\t\t\twarning(\"preferred mirror '%s' not listed \"\n+\t\t\t\t\t\"in remote.%s.mirror-url\",\n+\t\t\t\t\tremote->preferred_mirror,\n+\t\t\t\t\tremote->name);\n+\t\t\t}\n+\t\t}\n+\t\telse {\n+\t\t\tmirror_idx = 0;\n+\t\t}\n+\t}\n+\telse {\n+\t\tmirror_idx = remote_mirror_idx(remote, last_mirror) + 1;\n+\t\t// caller must check that we are not looping indefinitely\n+\t\tmirror_idx %= remote->mirror_url_nr;\n+\t}\n+\n+\turl = remote->mirror_url[mirror_idx];\n+\tret = transport_get(remote, url);\n+\n+\t// copy settings - caller must re-set options\n+\tret->verbose = transport->verbose;\n+\tret->progress = transport->progress;\n+\n+\treturn ret;\n+}\n+\n int transport_set_option(struct transport *transport,\n \t\t\t const char *name, const char *value)\n {\ndiff --git a/transport.h b/transport.h\nindex c14da6f..9890157 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -41,6 +41,9 @@ struct transport {\n /* Returns a transport suitable for the url */\n struct transport *transport_get(struct remote *, const char *);\n \n+/* Returns a transport for a mirror */\n+struct transport *transport_next_mirror(struct transport *transport, const char *last_mirror);\n+\n /* Transport options which apply to git:// and scp-style URLs */\n \n /* The program to use on the remote side to send a pack */\n@@ -78,6 +81,8 @@ int transport_fetch_refs(struct transport *transport, const struct ref *refs);\n void transport_unlock_pack(struct transport *transport);\n int transport_disconnect(struct transport *transport);\n char *transport_anonymize_url(const char *url);\n+struct refspec *mirror_refspec(struct transport* transport,\n+\t\t\t       struct refspec *refspec, int refspec_nr);\n \n /* Transport methods defined outside transport.c */\n int transport_helper_init(struct transport *transport, const char *name);\n-- \n1.6.3.3\n"},{"id":"128296","messageId":"1259143617-26580-4-git-send-email-sam@vilain.net","threadId":"21746","inReplyTo":"1259143617-26580-3-git-send-email-sam@vilain.net","subject":"[PATCH 3/4] fetch --use-mirror: don't fetch with 'autotags' for actual fetch","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-11-25T10:06:56Z","receivedAt":"2009-11-25T10:06:56Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Unsetting 'autotags' at a late stage during the fetch process has the\nuseful behaviour of figuring out which refs to fetch where according to the\nregular autotags rules, building a refspec (struct ref* linked list), and\nthen we turn them off for mirror fetch and no real tags are actually\nchanged, just the re-written ones under refs/mirrors/.  The final fetch\nwill re-set autotags again, and uncannily the exact behaviour we are after\nsprings up: we get all the tags for the refs that are now changing, even\nthough we got the data from a mirror.  All from one line of code.  Win!\n\nSigned-off-by: Sam Vilain <sam@vilain.net>\n---\n builtin-fetch.c         |    1 +\n t/t5560-mirror-fetch.sh |   12 +++++++++++-\n 2 files changed, 12 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin-fetch.c b/builtin-fetch.c\nindex b3b8766..daa287a 100644\n--- a/builtin-fetch.c\n+++ b/builtin-fetch.c\n@@ -269,6 +269,7 @@ static struct ref *get_ref_map(struct transport *transport,\n \t\tfind_non_local_tags(transport, &ref_map, &tail);\n \tref_remove_duplicates(ref_map);\n \tif (strcmp(transport->url, transport->remote->url[0]) != 0) {\n+\t\t*autotags = 0;\n \t\treturn mirror_refmap(transport, ref_map);\n \t}\n \ndiff --git a/t/t5560-mirror-fetch.sh b/t/t5560-mirror-fetch.sh\nindex 940dc0e..58d5f3c 100644\n--- a/t/t5560-mirror-fetch.sh\n+++ b/t/t5560-mirror-fetch.sh\n@@ -11,11 +11,13 @@ test_expect_success setup '\n \techo >file master initial &&\n \tgit add file &&\n \tgit commit -a -m \"Master initial\" &&\n+\tgit tag -m \"SEEN\" initial &&\n \tgit clone . master &&\n \tgit clone master mirror &&\n \tcd master &&\n \techo >file master update &&\n \tgit commit -a -m \"Master update\" &&\n+\tgit tag -m \"SEEN\" update &&\n \tcd .. &&\n \tmkdir clone &&\n \tcd clone &&\n@@ -35,12 +37,20 @@ test_expect_success 'fetch using mirror - explicit' '\n \n test_expect_success 'fetch using mirror - default' '\n \tcd .. &&\n+\tcd mirror &&\n+\tgit tag -m \"badtag\" badtag &&\n+\tcd .. &&\n \tmkdir clone2 &&\n \tcd clone2 &&\n \tgit init &&\n \tgit remote add origin ../master &&\n \tgit config remote.origin.mirror-url ../mirror\n \tgit fetch --use-mirror &&\n-\tgit rev-parse refs/mirrors/origin/localhost/heads/master\n+\tgit rev-parse refs/mirrors/origin/localhost/heads/master &&\n+\tgit rev-parse refs/mirrors/origin/localhost/tags/initial &&\n+\t! git rev-parse refs/tags/badtag &&\n+\tgit rev-parse refs/tags/initial &&\n+\tgit rev-parse refs/tags/update\n '\n+\n test_done\n-- \n1.6.3.3\n"},{"id":"128297","messageId":"1259143617-26580-5-git-send-email-sam@vilain.net","threadId":"21746","inReplyTo":"1259143617-26580-4-git-send-email-sam@vilain.net","subject":"[PATCH 4/4] fetch: cleanup refs with --use-mirror","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-11-25T10:06:57Z","receivedAt":"2009-11-25T10:06:57Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Remove identical refs after a successful fetch.  The ref under\n'refs/mirrors/HOST/XXX' is compared with 'refs/XXX', and if matched,\nthen the 'refs/mirrors/' version is removed.\n\nSigned-off-by: Sam Vilain <sam@vilain.net>\n---\n  This is a simple mechanism for removing stale mirror refs; a more\n  sophisticated approach would use the revision walker.\n\n builtin-fetch.c |   62 ++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 59 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin-fetch.c b/builtin-fetch.c\nindex daa287a..0c52f23 100644\n--- a/builtin-fetch.c\n+++ b/builtin-fetch.c\n@@ -211,6 +211,57 @@ struct ref *mirror_refmap(struct transport* transport,\n \treturn rv;\n }\n \n+int clean_up_mirror_ref(const char *refname,\n+\t\t\tconst unsigned char *sha1,\n+\t\t\tint flags,\n+\t\t\tvoid *leading)\n+{\n+\tchar *orig_refname;\n+\tchar *target_refname;\n+\tchar *x;\n+\tunsigned char found_sha1[20];\n+\n+\torig_refname = xmalloc(strlen(refname)+strlen(leading)+1);\n+\tx = strchr(refname, '/');\n+\tif (!x)\n+\t\treturn 0;\n+\ttarget_refname = xmalloc(strlen(x)+strlen(\"refs/remotes/\")+1); \n+\n+\tstrcpy(orig_refname, leading);\n+\tx = orig_refname + strlen(leading);\n+\tstrcpy(x, refname);\n+\n+\twarning(\"cleaning up mirror ref: %s (%s)\",\n+\t\torig_refname, sha1_to_hex(sha1));\n+\n+\tstrcpy(target_refname, \"refs/remotes/\");\n+\tstrcpy(target_refname+5, strchr(refname, '/')+1);\n+\n+\twarning(\"target ref is %s\", target_refname);\n+\n+\tif (resolve_ref(target_refname, found_sha1, 1, NULL)) {\n+\t\tif (!hashcmp(found_sha1, sha1)) {\n+\t\t\twarning(\"deleting ref %s\", orig_refname);\n+\t\t\tdelete_ref(orig_refname, sha1, REF_NODEREF);\n+\t\t}\n+\t}\n+}\n+\n+void clean_up_mirror_refs(struct remote* remote)\n+{\n+\tint rem_l = strlen(remote->name);\n+\tchar *dst_name = xmalloc(rem_l+14);\n+\tchar *x;\n+\tstrcpy(dst_name, \"refs/mirrors/\");\n+\tx = dst_name + 13;\n+\tstrcpy(x, remote->name);\n+\tx += rem_l;\n+\t*x++ = '/';\n+\n+\twarning(\"cleaning up mirror refs for remote %s\", remote->name);\n+\tfor_each_ref_in(dst_name, clean_up_mirror_ref,\n+\t\t\t(void *)dst_name);\n+}\n \n static struct ref *get_ref_map(struct transport *transport,\n \t\t\t       struct refspec *refs, int ref_count, int tags,\n@@ -884,9 +935,14 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\ttransport = NULL;\n \t\turls_remaining--;\n \t\tif (use_mirror) {\n-\t\t\tif (!exit_code && urls_remaining >= 1) {\n-\t\t\t\twarning(\"successful fetch from mirror\");\n-\t\t\t\turls_remaining = 1;\n+\t\t\tif (!exit_code) {\n+\t\t\t\tif (urls_remaining >= 1) {\n+\t\t\t\t\twarning(\"successful fetch from mirror\");\n+\t\t\t\t\turls_remaining = 1;\n+\t\t\t\t}\n+\t\t\t\telse {\n+\t\t\t\t\tclean_up_mirror_refs(remote);\n+\t\t\t\t}\n \t\t\t}\n \t\t\tif (urls_remaining == 1) {\n \t\t\t\ttransport = real_transport;\n-- \n1.6.3.3\n"},{"id":"128413","messageId":"20091126005837.GN11919@spearce.org","threadId":"21746","inReplyTo":"1259143617-26580-1-git-send-email-sam@vilain.net","subject":"Re: Client-side mirroring patches (v0)","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-11-26T00:58:37Z","receivedAt":"2009-11-26T00:58:37Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sam Vilain <sam@vilain.net> wrote:\n> Hey folks, this is the first stage of git mirroring \n...\n> Also there is the matter of falling over to the next mirror should one\n> not be reachable, but then we're getting into C weaknesses really.\n> Should I plan to do exception recovery using 'longjmp' ? \n\nPlease don't use longjmp.\n\nYou'll have to change the code to not die() upon connection failure,\nbut instead return an error code to the higher level which can\nlocate another mirror and retry.\n\n> Also the\n> process should be interruptible and provide a user menu.  Again this\n> seems like it would be very tedious and clumsy in C.  How do people\n> manage?\n\nWith great pain.  :-)\n\nTo do a user menu you can do a simple interface like `git add -i`\ndoes, which just dumps the choices to stdout and a prompt for the\nuser to enter their selection.  If you want something more complex\nyou need to link to curses or ncurses, which IIRC opens some issues\nwith portablity, but lets you do a bit nicer interface on the tty.\n\n-- \nShawn.\n"},{"id":"128419","messageId":"20091126012022.GP11919@spearce.org","threadId":"21746","inReplyTo":"1259143617-26580-3-git-send-email-sam@vilain.net","subject":"Re: [PATCH 2/4] fetch: try mirrors if selected","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-11-26T01:20:22Z","receivedAt":"2009-11-26T01:20:22Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sam Vilain <sam@vilain.net> wrote:\n> diff --git a/builtin-fetch.c b/builtin-fetch.c\n> index 209f502..b3b8766 100644\n> @@ -109,6 +109,109 @@ static void find_non_local_tags(struct transport *transport,\n>  \t\t\tstruct ref **head,\n>  \t\t\tstruct ref ***tail);\n>  \n> +char* get_url_hostname(const char *url)\n\nMinor nit, but we mark any function not used outside of the module\nas static.  Especially in a builtin-*.c since they all link into\nthe same namespace.  If this is meant to be reused, it belongs in\nconnect.c most likely, that's where we already have code like this\nto get the SSH hostname out of a URL for SSH connections.\n\nI don't have time right now to read the rest of this series, but\nthe general approach of fetching to a temporary mirror space before\nchecking if you really are current is a good one.\n\nI'm not sure that storing the list of mirrors inside of the remote\nmakes much sense, I would think the user would want to store only\na handful of \"fast\" URLs.\n\nAnd even then I wonder why this can't just be the url[1]..url[n-1]\nentries in the configuration file.  push pushes to all of the\nURLs at once, \"seeding the mirrors\".  Why can't fetch use the same\nconfiguration?\n\n-- \nShawn.\n"},{"id":"130643","messageId":"20100101090546.6117@nanako3.lavabit.com","threadId":"21746","inReplyTo":"1259143617-26580-1-git-send-email-sam@vilain.net","subject":"Re: Client-side mirroring patches (v0)","fromName":"Nanako Shiraishi","fromEmail":"nanako3@lavabit.com","sentAt":"2010-01-01T00:05:46Z","receivedAt":"2010-01-01T00:05:46Z","isPatch":false,"sender":{"key":"nanako3@lavabit.com","avatar":"https://gravatar.com/avatar/3777b9e201c5883a62b1a6fdf7c53f2d712d1d80989146063ea861e33aad72a8?d=mp&s=160"},"body":"Junio, could you tell us what happened to this thread?\n"}]}