{"thread":{"id":"23058","subject":"[PATCH 8/9] fetch: Don't fetch tags twice","startedAt":"2010-03-12T22:27:33Z","lastAt":"2010-03-18T19:42:49Z","messageCount":14,"participants":["Andreas Gruenbacher","Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"137086","messageId":"f747d7538f2a2bbdb48c9902298fa87691097eba.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":"f2bf11648d2cfb58e348d2c8caffa841012fd994.1268913163.git.agruen@suse.de","subject":"[PATCH 2/9] fetch: Properly initialize refspec on stack","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-12T22:27:33Z","receivedAt":"2010-03-12T22:27:33Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"Signed-off-by: Andreas Gruenbacher <agruen@suse.de>\n---\n builtin/fetch.c |    4 +---\n 1 files changed, 1 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 77031e2..660af84 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -107,10 +107,8 @@ static void add_merge_config(struct ref **head,\n \t\t * there is no entry in the resulting FETCH_HEAD marked\n \t\t * for merging.\n \t\t */\n+\t\tmemset(&refspec, 0, sizeof(refspec));\n \t\trefspec.src = branch->merge[i]->src;\n-\t\trefspec.dst = NULL;\n-\t\trefspec.pattern = 0;\n-\t\trefspec.force = 0;\n \t\tget_fetch_map(remote_refs, &refspec, tail, 1);\n \t\tfor (rm = *old_tail; rm; rm = rm->next)\n \t\t\trm->merge = 1;\n-- \n1.7.0.2.273.gc2413\n"},{"id":"137080","messageId":"f2bf11648d2cfb58e348d2c8caffa841012fd994.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":"cover.1268913163.git.agruen@suse.de","subject":"[PATCH 1/9] fetch: Check for a \"^{}\" suffix with suffixcmp()","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-13T17:17:04Z","receivedAt":"2010-03-13T17:17:04Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"Otherwise, we will check random bytes for ref names < 3 characters.\n\nSigned-off-by: Andreas Gruenbacher <agruen@suse.de>\n---\n builtin/fetch.c |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 2bb75c1..77031e2 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -590,7 +590,7 @@ static void find_non_local_tags(struct transport *transport,\n \t\t * to fetch then we can mark the ref entry in the list\n \t\t * as one to ignore by setting util to NULL.\n \t\t */\n-\t\tif (!strcmp(ref->name + strlen(ref->name) - 3, \"^{}\")) {\n+\t\tif (!suffixcmp(ref->name, \"^{}\")) {\n \t\t\tif (item && !has_sha1_file(ref->old_sha1) &&\n \t\t\t    !will_fetch(head, ref->old_sha1) &&\n \t\t\t    !has_sha1_file(item->util) &&\n-- \n1.7.0.2.273.gc2413\n"},{"id":"137085","messageId":"b7762c1e05412618584e38b868bcc870676f6e62.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":"f747d7538f2a2bbdb48c9902298fa87691097eba.1268913163.git.agruen@suse.de","subject":"[PATCH 3/9] fetch: Fix minor memory leak","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-15T22:18:48Z","receivedAt":"2010-03-15T22:18:48Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"A temporary struct ref is allocated in store_updated_refs() but not\nfreed.\n\nSigned-off-by: Andreas Gruenbacher <agruen@suse.de>\n---\n builtin/fetch.c |    5 +++--\n 1 files changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 660af84..957be9f 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -389,9 +389,10 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\t\t\tfputc(url[i], fp);\n \t\tfputc('\\n', fp);\n \n-\t\tif (ref)\n+\t\tif (ref) {\n \t\t\trc |= update_local_ref(ref, what, note);\n-\t\telse\n+\t\t\tfree(ref);\n+\t\t} else\n \t\t\tsprintf(note, \"* %-*s %-*s -> FETCH_HEAD\",\n \t\t\t\tTRANSPORT_SUMMARY_WIDTH, *kind ? kind : \"branch\",\n \t\t\t\t REFCOL_WIDTH, *what ? what : \"HEAD\");\n-- \n1.7.0.2.273.gc2413\n"},{"id":"137079","messageId":"672662f82e19ab268bd83f2b48980f43f126c089.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":"b7762c1e05412618584e38b868bcc870676f6e62.1268913163.git.agruen@suse.de","subject":"[PATCH 4/9] fetch: Move deepening fetch check into builtin/fetch.c","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-16T15:45:47Z","receivedAt":"2010-03-16T15:45:47Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"Check if we are doing a deepening fetch in fetch instead of guessing in\ntransport_fetch_refs().\n\nSigned-off-by: Andreas Gruenbacher <agruen@suse.de>\n---\n builtin/fetch.c |   34 ++++++++++++++++++++++++++++++++--\n cache.h         |    3 ++-\n transport.c     |   38 +++++++++++++-------------------------\n 3 files changed, 47 insertions(+), 28 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 957be9f..fc6efc7 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -489,11 +489,41 @@ static int quickfetch(struct ref *ref_map)\n \treturn finish_command(&revlist) || err;\n }\n \n+static void check_existing_refs(struct ref *ref_map) {\n+\tstruct ref *ref;\n+\tint done = 1;\n+\n+\tif (!depth) {\n+\t\tfor (ref = ref_map; ref; ref = ref->next) {\n+\t\t\tif (ref->peer_ref &&\n+\t\t\t    !is_null_sha1(ref->old_sha1) &&\n+\t\t\t    !hashcmp(ref->peer_ref->old_sha1, ref->old_sha1))\n+\t\t\t\tref->dont_fetch = 1;\n+\t\t\telse\n+\t\t\t\tdone = 0;\n+\t\t}\n+\t}\n+\tif (done) {\n+\t\t/*\n+\t\t * When deepening of a shallow repository is requested, local\n+\t\t * and remote refs are likely to still be equal.  In addition,\n+\t\t * we get here if all refs are up to date but quickfetch() did\n+\t\t * not succeed.  In both cases, fetch all refs.\n+\t\t */\n+\t\tfor (ref = ref_map; ref; ref = ref->next)\n+\t\t\tref->dont_fetch = 0;\n+\t}\n+}\n+\n static int fetch_refs(struct transport *transport, struct ref *ref_map)\n {\n-\tint ret = quickfetch(ref_map);\n-\tif (ret)\n+\tint ret;\n+\n+\tret = quickfetch(ref_map);\n+\tif (ret) {\n+\t\tcheck_existing_refs(ref_map);\n \t\tret = transport_fetch_refs(transport, ref_map);\n+\t}\n \tif (!ret)\n \t\tret |= store_updated_refs(transport->url,\n \t\t\t\ttransport->remote->name,\ndiff --git a/cache.h b/cache.h\nindex 89f6a40..abfc682 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -869,7 +869,8 @@ struct ref {\n \tunsigned int force:1,\n \t\tmerge:1,\n \t\tnonfastforward:1,\n-\t\tdeletion:1;\n+\t\tdeletion:1,\n+\t\tdont_fetch:1;\n \tenum {\n \t\tREF_STATUS_NONE = 0,\n \t\tREF_STATUS_OK,\ndiff --git a/transport.c b/transport.c\nindex 8ce3936..d1288af 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1127,36 +1127,24 @@ const struct ref *transport_get_remote_refs(struct transport *transport)\n \n int transport_fetch_refs(struct transport *transport, struct ref *refs)\n {\n-\tint rc;\n-\tint nr_heads = 0, nr_alloc = 0, nr_refs = 0;\n-\tstruct ref **heads = NULL;\n+\tint rc = 0;\n+\tint nr_heads;\n+\tstruct ref **heads;\n \tstruct ref *rm;\n \n-\tfor (rm = refs; rm; rm = rm->next) {\n-\t\tnr_refs++;\n-\t\tif (rm->peer_ref &&\n-\t\t    !is_null_sha1(rm->old_sha1) &&\n-\t\t    !hashcmp(rm->peer_ref->old_sha1, rm->old_sha1))\n+\tfor (nr_heads = 0, rm = refs; rm; rm = rm->next) {\n+\t\tif (rm->dont_fetch)\n \t\t\tcontinue;\n-\t\tALLOC_GROW(heads, nr_heads + 1, nr_alloc);\n-\t\theads[nr_heads++] = rm;\n+\t\tnr_heads++;\n \t}\n-\n-\tif (!nr_heads) {\n-\t\t/*\n-\t\t * When deepening of a shallow repository is requested,\n-\t\t * then local and remote refs are likely to still be equal.\n-\t\t * Just feed them all to the fetch method in that case.\n-\t\t * This condition shouldn't be met in a non-deepening fetch\n-\t\t * (see builtin-fetch.c:quickfetch()).\n-\t\t */\n-\t\theads = xmalloc(nr_refs * sizeof(*heads));\n-\t\tfor (rm = refs; rm; rm = rm->next)\n-\t\t\theads[nr_heads++] = rm;\n+\theads = xmalloc(nr_heads * sizeof(*heads));\n+\tfor (nr_heads = 0, rm = refs; rm; rm = rm->next) {\n+\t\tif (rm->dont_fetch)\n+\t\t\tcontinue;\n+\t\theads[nr_heads++] = rm;\n \t}\n-\n-\trc = transport->fetch(transport, nr_heads, heads);\n-\n+\tif (nr_heads)\n+\t\trc = transport->fetch(transport, nr_heads, heads);\n \tfree(heads);\n \treturn rc;\n }\n-- \n1.7.0.2.273.gc2413\n"},{"id":"137084","messageId":"dbb0f92deca35a16c898af697dc41affc377ae0b.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":"672662f82e19ab268bd83f2b48980f43f126c089.1268913163.git.agruen@suse.de","subject":"[PATCH 5/9] fetch: Move loop checking which refs we have already","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-16T17:49:32Z","receivedAt":"2010-03-16T17:49:32Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"This loop fits better into get_ref_map(); move it there.\n\nSigned-off-by: Andreas Gruenbacher <agruen@suse.de>\n---\n builtin/fetch.c |   48 ++++++++++++++++++++++++------------------------\n 1 files changed, 24 insertions(+), 24 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex fc6efc7..6dc75d4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -115,6 +115,15 @@ static void add_merge_config(struct ref **head,\n \t}\n }\n \n+static int add_existing(const char *refname, const unsigned char *sha1,\n+\t\t\tint flag, void *cbdata)\n+{\n+\tstruct string_list *list = (struct string_list *)cbdata;\n+\tstruct string_list_item *item = string_list_insert(refname, list);\n+\titem->util = (void *)sha1;\n+\treturn 0;\n+}\n+\n static void find_non_local_tags(struct transport *transport,\n \t\t\tstruct ref **head,\n \t\t\tstruct ref ***tail);\n@@ -123,6 +132,7 @@ static struct ref *get_ref_map(struct transport *transport,\n \t\t\t       struct refspec *refs, int ref_count, int tags,\n \t\t\t       int *autotags)\n {\n+\tstruct string_list existing_refs = { NULL, 0, 0, 0 };\n \tint i;\n \tstruct ref *rm;\n \tstruct ref *ref_map = NULL;\n@@ -176,6 +186,20 @@ static struct ref *get_ref_map(struct transport *transport,\n \t\tfind_non_local_tags(transport, &ref_map, &tail);\n \tref_remove_duplicates(ref_map);\n \n+\tfor_each_ref(add_existing, &existing_refs);\n+\tfor (rm = ref_map; rm; rm = rm->next) {\n+\t\tif (rm->peer_ref) {\n+\t\t\tstruct string_list_item *peer_item;\n+\n+\t\t\tpeer_item = string_list_lookup(rm->peer_ref->name,\n+\t\t\t\t\t\t       &existing_refs);\n+\t\t\tif (peer_item)\n+\t\t\t\thashcpy(rm->peer_ref->old_sha1,\n+\t\t\t\t\tpeer_item->util);\n+\t\t}\n+\t}\n+\tstring_list_clear(&existing_refs, 0);\n+\n \treturn ref_map;\n }\n \n@@ -554,15 +578,6 @@ static int prune_refs(struct transport *transport, struct ref *ref_map)\n \treturn result;\n }\n \n-static int add_existing(const char *refname, const unsigned char *sha1,\n-\t\t\tint flag, void *cbdata)\n-{\n-\tstruct string_list *list = (struct string_list *)cbdata;\n-\tstruct string_list_item *item = string_list_insert(refname, list);\n-\titem->util = (void *)sha1;\n-\treturn 0;\n-}\n-\n static int will_fetch(struct ref **head, const unsigned char *sha1)\n {\n \tstruct ref *rm = *head;\n@@ -696,14 +711,9 @@ static int truncate_fetch_head(void)\n static int do_fetch(struct transport *transport,\n \t\t    struct refspec *refs, int ref_count)\n {\n-\tstruct string_list existing_refs = { NULL, 0, 0, 0 };\n-\tstruct string_list_item *peer_item = NULL;\n \tstruct ref *ref_map;\n-\tstruct ref *rm;\n \tint autotags = (transport->remote->fetch_tags == 1);\n \n-\tfor_each_ref(add_existing, &existing_refs);\n-\n \tif (transport->remote->fetch_tags == 2 && tags != TAGS_UNSET)\n \t\ttags = TAGS_SET;\n \tif (transport->remote->fetch_tags == -1)\n@@ -723,16 +733,6 @@ static int do_fetch(struct transport *transport,\n \tif (!update_head_ok)\n \t\tcheck_not_current_branch(ref_map);\n \n-\tfor (rm = ref_map; rm; rm = rm->next) {\n-\t\tif (rm->peer_ref) {\n-\t\t\tpeer_item = string_list_lookup(rm->peer_ref->name,\n-\t\t\t\t\t\t       &existing_refs);\n-\t\t\tif (peer_item)\n-\t\t\t\thashcpy(rm->peer_ref->old_sha1,\n-\t\t\t\t\tpeer_item->util);\n-\t\t}\n-\t}\n-\n \tif (tags == TAGS_DEFAULT && autotags)\n \t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, \"1\");\n \tif (fetch_refs(transport, ref_map)) {\n-- \n1.7.0.2.273.gc2413\n"},{"id":"137082","messageId":"d3c00e2a27003dca196d5480007544610cc1e5b8.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":"dbb0f92deca35a16c898af697dc41affc377ae0b.1268913163.git.agruen@suse.de","subject":"[PATCH 6/9] fetch: Check if all objects exist after fetching","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-16T18:48:07Z","receivedAt":"2010-03-16T18:48:07Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"Check if all objects reachable from the fetched refs exist after\nfetching instead of before: this allows us to distinguish between a\nrepository which is not up to date and a corrupted repository, and to\nensure that the repository is up to date and complete after the fetch.\n\nIn select_missing_refs() we must check which objects we have, instead of\nchecking which refs are up to date: otherwise, we might fetch objects\ninto the repository which exist in an alternate.\n\nSigned-off-by: Andreas Gruenbacher <agruen@suse.de>\n---\n builtin/fetch.c |   82 ++++++++++++++++++++++++++-----------------------------\n 1 files changed, 39 insertions(+), 43 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 6dc75d4..0f292b8 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -440,20 +440,15 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n }\n \n /*\n- * We would want to bypass the object transfer altogether if\n- * everything we are going to fetch already exists and is connected\n- * locally.\n- *\n- * The refs we are going to fetch are in ref_map.  If running\n+ * If running\n  *\n  *  $ git rev-list --objects --stdin --not --all\n  *\n  * (feeding all the refs in ref_map on its standard input)\n  * does not error out, that means everything reachable from the\n- * refs we are going to fetch exists and is connected to some of\n- * our existing refs.\n+ * refs we have fetched exists.\n  */\n-static int quickfetch(struct ref *ref_map)\n+static int check_connectivity(struct ref *ref_map)\n {\n \tstruct child_process revlist;\n \tstruct ref *ref;\n@@ -461,16 +456,6 @@ static int quickfetch(struct ref *ref_map)\n \tconst char *argv[] = {\"rev-list\",\n \t\t\"--quiet\", \"--objects\", \"--stdin\", \"--not\", \"--all\", NULL};\n \n-\t/*\n-\t * If we are deepening a shallow clone we already have these\n-\t * objects reachable.  Running rev-list here will return with\n-\t * a good (0) exit status and we'll bypass the fetch that we\n-\t * really need to perform.  Claiming failure now will ensure\n-\t * we perform the network exchange to deepen our history.\n-\t */\n-\tif (depth)\n-\t\treturn -1;\n-\n \tif (!ref_map)\n \t\treturn 0;\n \n@@ -513,46 +498,57 @@ static int quickfetch(struct ref *ref_map)\n \treturn finish_command(&revlist) || err;\n }\n \n-static void check_existing_refs(struct ref *ref_map) {\n+static void select_missing_refs(struct ref *ref_map) {\n \tstruct ref *ref;\n-\tint done = 1;\n \n-\tif (!depth) {\n-\t\tfor (ref = ref_map; ref; ref = ref->next) {\n-\t\t\tif (ref->peer_ref &&\n-\t\t\t    !is_null_sha1(ref->old_sha1) &&\n-\t\t\t    !hashcmp(ref->peer_ref->old_sha1, ref->old_sha1))\n-\t\t\t\tref->dont_fetch = 1;\n-\t\t\telse\n-\t\t\t\tdone = 0;\n-\t\t}\n+\tfor (ref = ref_map; ref; ref = ref->next) {\n+\t\tif (has_sha1_file(ref->old_sha1))\n+\t\t\tref->dont_fetch = 1;\n \t}\n-\tif (done) {\n-\t\t/*\n-\t\t * When deepening of a shallow repository is requested, local\n-\t\t * and remote refs are likely to still be equal.  In addition,\n-\t\t * we get here if all refs are up to date but quickfetch() did\n-\t\t * not succeed.  In both cases, fetch all refs.\n-\t\t */\n-\t\tfor (ref = ref_map; ref; ref = ref->next)\n+}\n+\n+static int select_all_refs(struct ref *ref_map) {\n+\tstruct ref *ref;\n+\tint done = 1;\n+\n+\tfor (ref = ref_map; ref; ref = ref->next) {\n+\t\tif (ref->dont_fetch) {\n \t\t\tref->dont_fetch = 0;\n+\t\t\tdone = 0;\n+\t\t}\n \t}\n+\n+\treturn done;\n }\n \n static int fetch_refs(struct transport *transport, struct ref *ref_map)\n {\n \tint ret;\n \n-\tret = quickfetch(ref_map);\n+\tif (depth) {\n+\t\t/* When deepening a shallow clone, re-fetch all the refs:\n+\t\t * it doesn't help to have the topmost commits of each ref.\n+\t\t */\n+\t\tselect_all_refs(ref_map);\n+\t} else\n+\t\tselect_missing_refs(ref_map);\n+\tret = transport_fetch_refs(transport, ref_map);\n+\tif (ret)\n+\t\treturn ret;\n+\tret = check_connectivity(ref_map);\n \tif (ret) {\n-\t\tcheck_existing_refs(ref_map);\n+\t\tif (select_all_refs(ref_map))\n+\t\t\treturn ret;\n+\t\ttransport_unlock_pack(transport);\n \t\tret = transport_fetch_refs(transport, ref_map);\n+\t\tif (ret)\n+\t\t\treturn ret;\n \t}\n-\tif (!ret)\n-\t\tret |= store_updated_refs(transport->url,\n-\t\t\t\ttransport->remote->name,\n-\t\t\t\tref_map);\n+\tret = store_updated_refs(transport->url,\n+\t\t\t\t transport->remote->name,\n+\t\t\t\t ref_map);\n \ttransport_unlock_pack(transport);\n+\n \treturn ret;\n }\n \n-- \n1.7.0.2.273.gc2413\n"},{"id":"137087","messageId":"24bf0bf2a12b1eccf3aa8f6bd1753012dfdb9987.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":"d3c00e2a27003dca196d5480007544610cc1e5b8.1268913163.git.agruen@suse.de","subject":"[PATCH 7/9] fetch: Use the same ref map for all branches and tags","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-17T21:42:27Z","receivedAt":"2010-03-17T21:42:27Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"With automatic tag following and the standard refspec for remotes\n(+refs/heads/*:refs/remotes/<remote>/*), we fetch all branches and tags\nthat we know we need, and then we check which additional tags have\nbecome reachable in a separate second pass.  The refspec used for tags\nis always hardcoded (+refs/tags/*:refs/tags/*).\n\nChange this to compute the mapping between remote and peer refs only\nonce, and flag tags which may become reachable as \"automatic\".  After\nthe first pass, go through all \"automatic\" tags in the mapping to see\nwhich have actually become reachable, and process these.\n\nThis change preserves the existing behavior, but allows to subsequently\nfix the handling of refspecs which match tags, like\n+refs/tags/*:refs/tags/<remote>/*.\n\nSigned-off-by: Andreas Gruenbacher <agruen@suse.de>\n---\n builtin/fetch.c |  286 +++++++++++++++++++++++++++++--------------------------\n cache.h         |    3 +-\n 2 files changed, 155 insertions(+), 134 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 0f292b8..8728153 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -124,12 +124,97 @@ static int add_existing(const char *refname, const unsigned char *sha1,\n \treturn 0;\n }\n \n-static void find_non_local_tags(struct transport *transport,\n-\t\t\tstruct ref **head,\n-\t\t\tstruct ref ***tail);\n+static void move_tag_to_tail(struct ref **tag, struct ref ***tail) {\n+\tstruct ref *ref = *tag;\n+\n+\t*tag = ref->next;\n+\t**tail = ref;\n+\t*tail = &ref->next;\n+\tref->next = NULL;\n+}\n+\n+static int will_fetch(struct ref *ref_map, const unsigned char *sha1)\n+{\n+\tstruct ref *ref;\n+\n+\tfor (ref = ref_map; ref; ref = ref->next)\n+\t\tif (!hashcmp(ref->old_sha1, sha1))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static void add_default_tags(const struct ref *remote_refs,\n+\t\t\t     struct string_list *existing_refs,\n+\t\t\t     struct string_list *peeled_map,\n+\t\t\t     struct ref *ref_map,\n+\t\t\t     struct ref ***tail)\n+{\n+\tstruct ref *tags = NULL, **tags_tail = &tags;\n+\tstruct ref *ref, *peer_ref, **tag;\n+\n+\tfor (ref = (struct ref *)remote_refs; ref; ref = ref->next) {\n+\t\tint len;\n+\n+\t\tif (!ref->next)\n+\t\t\tbreak;\n+\t\tlen = strlen(ref->name);\n+\t\tif (!strncmp(ref->name, ref->next->name, len) &&\n+\t\t    !strcmp(ref->next->name + len, \"^{}\")) {\n+\t\t\tstruct string_list_item *item;\n+\n+\t\t\titem = string_list_append(ref->name, peeled_map);\n+\t\t\titem->util = ref->next->old_sha1;\n+\t\t}\n+\t}\n+\tsort_string_list(peeled_map);\n+\n+\tfor (ref = ref_map; ref; ref = ref->next)\n+\t\tif (ref->peer_ref)\n+\t\t\tstring_list_insert(ref->peer_ref->name, existing_refs);\n+\n+\tget_fetch_map(remote_refs, tag_refspec, &tags_tail, 0);\n+\tfor (tag = &tags; *tag; ) {\n+\t\tref = *tag;\n+\t\tpeer_ref = ref->peer_ref;\n+\t\tif (!string_list_has_string(existing_refs, peer_ref->name)) {\n+\t\t\tstruct string_list_item *item;\n+\n+\t\t\titem = string_list_lookup(ref->name, peeled_map);\n+\t\t\tref->automatic = !(item && will_fetch(ref_map, item->util));\n+\t\t\tstring_list_insert(peer_ref->name, existing_refs);\n+\t\t\tmove_tag_to_tail(tag, tail);\n+\t\t\tcontinue;\n+\t\t}\n+\t\ttag = &(*tag)->next;\n+\t}\n+\n+\tfree_refs(tags);\n+}\n+\n+static void discard_refs(struct ref **ref_map, int not, int (*fn)(const struct ref *)) {\n+\tstruct ref *discard = NULL;\n+\n+\twhile (*ref_map) {\n+\t\tstruct ref *ref = *ref_map;\n+\n+\t\tif (!not ^ !fn(ref)) {\n+\t\t\t*ref_map = ref->next;\n+\t\t\tref->next = discard;\n+\t\t\tdiscard = ref;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tref_map = &ref->next;\n+\t}\n+\tfree_refs(discard);\n+}\n+\n+int ref_is_automatic(const struct ref *ref) {\n+\treturn ref->automatic;\n+}\n \n static struct ref *get_ref_map(struct transport *transport,\n \t\t\t       struct refspec *refs, int ref_count, int tags,\n+\t\t\t       struct string_list *peeled_map,\n \t\t\t       int *autotags)\n {\n \tstruct string_list existing_refs = { NULL, 0, 0, 0 };\n@@ -182,9 +267,6 @@ static struct ref *get_ref_map(struct transport *transport,\n \t\t\ttail = &ref_map->next;\n \t\t}\n \t}\n-\tif (tags == TAGS_DEFAULT && *autotags)\n-\t\tfind_non_local_tags(transport, &ref_map, &tail);\n-\tref_remove_duplicates(ref_map);\n \n \tfor_each_ref(add_existing, &existing_refs);\n \tfor (rm = ref_map; rm; rm = rm->next) {\n@@ -198,8 +280,13 @@ static struct ref *get_ref_map(struct transport *transport,\n \t\t\t\t\tpeer_item->util);\n \t\t}\n \t}\n+\n+\tif (tags == TAGS_DEFAULT && *autotags)\n+\t\tadd_default_tags(remote_refs, &existing_refs, peeled_map, ref_map, &tail);\n \tstring_list_clear(&existing_refs, 0);\n \n+\tref_remove_duplicates(ref_map);\n+\n \treturn ref_map;\n }\n \n@@ -353,12 +440,22 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tfor (rm = ref_map; rm; rm = rm->next) {\n \t\tstruct ref *ref = NULL;\n \n+\t\tif (rm->automatic) {\n+\t\t\tif (has_sha1_file(rm->old_sha1))\n+\t\t\t\trm->automatic = 0;\n+\t\t\telse\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (rm->peer_ref) {\n \t\t\tref = xcalloc(1, sizeof(*ref) + strlen(rm->peer_ref->name) + 1);\n \t\t\tstrcpy(ref->name, rm->peer_ref->name);\n \t\t\thashcpy(ref->old_sha1, rm->peer_ref->old_sha1);\n \t\t\thashcpy(ref->new_sha1, rm->old_sha1);\n \t\t\tref->force = rm->peer_ref->force;\n+\n+\t\t\t/* Remember we have updated this ref */\n+\t\t\thashcpy(rm->peer_ref->old_sha1, rm->old_sha1);\n \t\t}\n \n \t\tcommit = lookup_commit_reference_gently(rm->old_sha1, 1);\n@@ -439,6 +536,15 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \treturn rc;\n }\n \n+static int contains_only_automatic_refs(struct ref *ref_map) {\n+\tstruct ref *ref;\n+\n+\tfor (ref = ref_map; ref; ref = ref->next)\n+\t\tif (!ref->automatic)\n+\t\t\treturn 0;\n+\treturn 1;\n+}\n+\n /*\n  * If running\n  *\n@@ -456,7 +562,7 @@ static int check_connectivity(struct ref *ref_map)\n \tconst char *argv[] = {\"rev-list\",\n \t\t\"--quiet\", \"--objects\", \"--stdin\", \"--not\", \"--all\", NULL};\n \n-\tif (!ref_map)\n+\tif (contains_only_automatic_refs(ref_map))\n \t\treturn 0;\n \n \tmemset(&revlist, 0, sizeof(revlist));\n@@ -479,6 +585,9 @@ static int check_connectivity(struct ref *ref_map)\n \tsigchain_push(SIGPIPE, SIG_IGN);\n \n \tfor (ref = ref_map; ref; ref = ref->next) {\n+\t\tif (ref->automatic)\n+\t\t\tcontinue;\n+\n \t\tif (write_in_full(revlist.in, sha1_to_hex(ref->old_sha1), 40) < 0 ||\n \t\t    write_str_in_full(revlist.in, \"\\n\") < 0) {\n \t\t\tif (errno != EPIPE && errno != EINVAL)\n@@ -501,10 +610,9 @@ static int check_connectivity(struct ref *ref_map)\n static void select_missing_refs(struct ref *ref_map) {\n \tstruct ref *ref;\n \n-\tfor (ref = ref_map; ref; ref = ref->next) {\n-\t\tif (has_sha1_file(ref->old_sha1))\n-\t\t\tref->dont_fetch = 1;\n-\t}\n+\tfor (ref = ref_map; ref; ref = ref->next)\n+\t\tref->dont_fetch = ref->automatic ||\n+\t\t\t\t  has_sha1_file(ref->old_sha1);\n }\n \n static int select_all_refs(struct ref *ref_map) {\n@@ -512,7 +620,7 @@ static int select_all_refs(struct ref *ref_map) {\n \tint done = 1;\n \n \tfor (ref = ref_map; ref; ref = ref->next) {\n-\t\tif (ref->dont_fetch) {\n+\t\tif (ref->dont_fetch && !ref->automatic) {\n \t\t\tref->dont_fetch = 0;\n \t\t\tdone = 0;\n \t\t}\n@@ -574,111 +682,6 @@ static int prune_refs(struct transport *transport, struct ref *ref_map)\n \treturn result;\n }\n \n-static int will_fetch(struct ref **head, const unsigned char *sha1)\n-{\n-\tstruct ref *rm = *head;\n-\twhile (rm) {\n-\t\tif (!hashcmp(rm->old_sha1, sha1))\n-\t\t\treturn 1;\n-\t\trm = rm->next;\n-\t}\n-\treturn 0;\n-}\n-\n-struct tag_data {\n-\tstruct ref **head;\n-\tstruct ref ***tail;\n-};\n-\n-static int add_to_tail(struct string_list_item *item, void *cb_data)\n-{\n-\tstruct tag_data *data = (struct tag_data *)cb_data;\n-\tstruct ref *rm = NULL;\n-\n-\t/* We have already decided to ignore this item */\n-\tif (!item->util)\n-\t\treturn 0;\n-\n-\trm = alloc_ref(item->string);\n-\trm->peer_ref = alloc_ref(item->string);\n-\thashcpy(rm->old_sha1, item->util);\n-\n-\t**data->tail = rm;\n-\t*data->tail = &rm->next;\n-\n-\treturn 0;\n-}\n-\n-static void find_non_local_tags(struct transport *transport,\n-\t\t\tstruct ref **head,\n-\t\t\tstruct ref ***tail)\n-{\n-\tstruct string_list existing_refs = { NULL, 0, 0, 0 };\n-\tstruct string_list remote_refs = { NULL, 0, 0, 0 };\n-\tstruct tag_data data = {head, tail};\n-\tconst struct ref *ref;\n-\tstruct string_list_item *item = NULL;\n-\n-\tfor_each_ref(add_existing, &existing_refs);\n-\tfor (ref = transport_get_remote_refs(transport); ref; ref = ref->next) {\n-\t\tif (prefixcmp(ref->name, \"refs/tags\"))\n-\t\t\tcontinue;\n-\n-\t\t/*\n-\t\t * The peeled ref always follows the matching base\n-\t\t * ref, so if we see a peeled ref that we don't want\n-\t\t * to fetch then we can mark the ref entry in the list\n-\t\t * as one to ignore by setting util to NULL.\n-\t\t */\n-\t\tif (!suffixcmp(ref->name, \"^{}\")) {\n-\t\t\tif (item && !has_sha1_file(ref->old_sha1) &&\n-\t\t\t    !will_fetch(head, ref->old_sha1) &&\n-\t\t\t    !has_sha1_file(item->util) &&\n-\t\t\t    !will_fetch(head, item->util))\n-\t\t\t\titem->util = NULL;\n-\t\t\titem = NULL;\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\t/*\n-\t\t * If item is non-NULL here, then we previously saw a\n-\t\t * ref not followed by a peeled reference, so we need\n-\t\t * to check if it is a lightweight tag that we want to\n-\t\t * fetch.\n-\t\t */\n-\t\tif (item && !has_sha1_file(item->util) &&\n-\t\t    !will_fetch(head, item->util))\n-\t\t\titem->util = NULL;\n-\n-\t\titem = NULL;\n-\n-\t\t/* skip duplicates and refs that we already have */\n-\t\tif (string_list_has_string(&remote_refs, ref->name) ||\n-\t\t    string_list_has_string(&existing_refs, ref->name))\n-\t\t\tcontinue;\n-\n-\t\titem = string_list_insert(ref->name, &remote_refs);\n-\t\titem->util = (void *)ref->old_sha1;\n-\t}\n-\tstring_list_clear(&existing_refs, 0);\n-\n-\t/*\n-\t * We may have a final lightweight tag that needs to be\n-\t * checked to see if it needs fetching.\n-\t */\n-\tif (item && !has_sha1_file(item->util) &&\n-\t    !will_fetch(head, item->util))\n-\t\titem->util = NULL;\n-\n-\t/*\n-\t * For all the tags in the remote_refs string list, call\n-\t * add_to_tail to add them to the list of refs to be fetched\n-\t */\n-\tfor_each_string_list(add_to_tail, &remote_refs, &data);\n-\n-\tstring_list_clear(&remote_refs, 0);\n-}\n-\n static void check_not_current_branch(struct ref *ref_map)\n {\n \tstruct branch *current_branch = branch_get(NULL);\n@@ -704,9 +707,29 @@ static int truncate_fetch_head(void)\n \treturn 0;\n }\n \n+static int need_additional_refs(struct ref *ref_map, struct string_list *peeled_map) {\n+\tint additional_refs = 0;\n+\tstruct ref *ref;\n+\n+\tfor (ref = ref_map; ref; ref = ref->next) {\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (!ref->automatic)\n+\t\t\tcontinue;\n+\t\titem = string_list_lookup(ref->name, peeled_map);\n+\t\tif (item && has_sha1_file(item->util)) {\n+\t\t\tref->automatic = 0;\n+\t\t\tadditional_refs = 1;\n+\t\t}\n+\t}\n+\n+\treturn additional_refs;\n+}\n+\n static int do_fetch(struct transport *transport,\n \t\t    struct refspec *refs, int ref_count)\n {\n+\tstruct string_list peeled_map = { NULL, 0, 0, 0 };\n \tstruct ref *ref_map;\n \tint autotags = (transport->remote->fetch_tags == 1);\n \n@@ -725,7 +748,7 @@ static int do_fetch(struct transport *transport,\n \t\t\treturn errcode;\n \t}\n \n-\tref_map = get_ref_map(transport, refs, ref_count, tags, &autotags);\n+\tref_map = get_ref_map(transport, refs, ref_count, tags, &peeled_map, &autotags);\n \tif (!update_head_ok)\n \t\tcheck_not_current_branch(ref_map);\n \n@@ -735,23 +758,20 @@ static int do_fetch(struct transport *transport,\n \t\tfree_refs(ref_map);\n \t\treturn 1;\n \t}\n-\tif (prune)\n-\t\tprune_refs(transport, ref_map);\n-\tfree_refs(ref_map);\n-\n-\t/* if neither --no-tags nor --tags was specified, do automated tag\n-\t * following ... */\n-\tif (tags == TAGS_DEFAULT && autotags) {\n-\t\tstruct ref **tail = &ref_map;\n-\t\tref_map = NULL;\n-\t\tfind_non_local_tags(transport, &ref_map, &tail);\n-\t\tif (ref_map) {\n-\t\t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, NULL);\n-\t\t\ttransport_set_option(transport, TRANS_OPT_DEPTH, \"0\");\n-\t\t\tfetch_refs(transport, ref_map);\n+\tif (need_additional_refs(ref_map, &peeled_map)) {\n+\t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, NULL);\n+\t\ttransport_set_option(transport, TRANS_OPT_DEPTH, \"0\");\n+\t\tif (fetch_refs(transport, ref_map)) {\n+\t\t\tfree_refs(ref_map);\n+\t\t\treturn 1;\n \t\t}\n-\t\tfree_refs(ref_map);\n \t}\n+\tif (prune) {\n+\t\tdiscard_refs(&ref_map, 0, ref_is_automatic);\n+\t\tprune_refs(transport, ref_map);\n+\t}\n+\tstring_list_clear(&peeled_map, 0);\n+\tfree_refs(ref_map);\n \n \treturn 0;\n }\ndiff --git a/cache.h b/cache.h\nindex abfc682..1aa3000 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -870,7 +870,8 @@ struct ref {\n \t\tmerge:1,\n \t\tnonfastforward:1,\n \t\tdeletion:1,\n-\t\tdont_fetch:1;\n+\t\tdont_fetch:1,\n+\t\tautomatic:1;\n \tenum {\n \t\tREF_STATUS_NONE = 0,\n \t\tREF_STATUS_OK,\n-- \n1.7.0.2.273.gc2413\n"},{"id":"137078","messageId":"3f1d446c333b7eaabce513e846efe5313d2084fb.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":"24bf0bf2a12b1eccf3aa8f6bd1753012dfdb9987.1268913163.git.agruen@suse.de","subject":"[PATCH 8/9] fetch: Don't fetch tags twice","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-17T23:02:23Z","receivedAt":"2010-03-17T23:02:23Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"When refspecs are fetched which match tags (e.g.,\n+refs/tags/*:refs/tags/<remote>/* or +refs/*:refs/remotes/<remote>/*)\nand without --no-tags, we end up fetching the same tags twice because of\nthe +refs/tags/*:refs/tags/* refspec built into fetch.  Instead, when\nprocessing the built-in refspec, ignore tags which are already mapped.\n\nSigned-off-by: Andreas Gruenbacher <agruen@suse.de>\n---\n builtin/fetch.c |   10 ++++++++--\n 1 files changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 8728153..8c01876 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -149,6 +149,7 @@ static void add_default_tags(const struct ref *remote_refs,\n \t\t\t     struct ref *ref_map,\n \t\t\t     struct ref ***tail)\n {\n+\tstruct string_list mapped_refs = { NULL, 0, 0, 0 };\n \tstruct ref *tags = NULL, **tags_tail = &tags;\n \tstruct ref *ref, *peer_ref, **tag;\n \n@@ -168,15 +169,19 @@ static void add_default_tags(const struct ref *remote_refs,\n \t}\n \tsort_string_list(peeled_map);\n \n-\tfor (ref = ref_map; ref; ref = ref->next)\n+\tfor (ref = ref_map; ref; ref = ref->next) {\n+\t\tstring_list_append(ref->name, &mapped_refs);\n \t\tif (ref->peer_ref)\n \t\t\tstring_list_insert(ref->peer_ref->name, existing_refs);\n+\t}\n+\tsort_string_list(&mapped_refs);\n \n \tget_fetch_map(remote_refs, tag_refspec, &tags_tail, 0);\n \tfor (tag = &tags; *tag; ) {\n \t\tref = *tag;\n \t\tpeer_ref = ref->peer_ref;\n-\t\tif (!string_list_has_string(existing_refs, peer_ref->name)) {\n+\t\tif (!(string_list_has_string(existing_refs, peer_ref->name) ||\n+\t\t      string_list_has_string(&mapped_refs, ref->name))) {\n \t\t\tstruct string_list_item *item;\n \n \t\t\titem = string_list_lookup(ref->name, peeled_map);\n@@ -187,6 +192,7 @@ static void add_default_tags(const struct ref *remote_refs,\n \t\t}\n \t\ttag = &(*tag)->next;\n \t}\n+\tstring_list_clear(&mapped_refs, 0);\n \n \tfree_refs(tags);\n }\n-- \n1.7.0.2.273.gc2413\n"},{"id":"137083","messageId":"ff287641841582e06f6a145e9c697badb089b9d4.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":"3f1d446c333b7eaabce513e846efe5313d2084fb.1268913163.git.agruen@suse.de","subject":"[PATCH 9/9] fetch: Make automatic tag following work with arbitrary refspecs","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-17T23:59:22Z","receivedAt":"2010-03-17T23:59:22Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"Make automatic tag following and the --tags and --no-tags options work\ntogether with arbitrary refspecs: by default, tags will always be\nfetched automatically, the --tags option will fetch all tags (and only\ntags), and the --no-tags option will not fetch any tags.  This allows to\ndefine how things are mapped and which things to fetch independently.\n\nSigned-off-by: Andres Gruenbacher <agruen@suse.de>\n---\n builtin/fetch.c |   37 ++++++++++++++++++++++++++-----------\n 1 files changed, 26 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 8c01876..9265d76 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -147,6 +147,7 @@ static void add_default_tags(const struct ref *remote_refs,\n \t\t\t     struct string_list *existing_refs,\n \t\t\t     struct string_list *peeled_map,\n \t\t\t     struct ref *ref_map,\n+\t\t\t     int overwrite,\n \t\t\t     struct ref ***tail)\n {\n \tstruct string_list mapped_refs = { NULL, 0, 0, 0 };\n@@ -180,12 +181,8 @@ static void add_default_tags(const struct ref *remote_refs,\n \tfor (tag = &tags; *tag; ) {\n \t\tref = *tag;\n \t\tpeer_ref = ref->peer_ref;\n-\t\tif (!(string_list_has_string(existing_refs, peer_ref->name) ||\n-\t\t      string_list_has_string(&mapped_refs, ref->name))) {\n-\t\t\tstruct string_list_item *item;\n-\n-\t\t\titem = string_list_lookup(ref->name, peeled_map);\n-\t\t\tref->automatic = !(item && will_fetch(ref_map, item->util));\n+\t\tif ((overwrite || !string_list_has_string(existing_refs, peer_ref->name)) &&\n+\t\t    !string_list_has_string(&mapped_refs, ref->name)) {\n \t\t\tstring_list_insert(peer_ref->name, existing_refs);\n \t\t\tmove_tag_to_tail(tag, tail);\n \t\t\tcontinue;\n@@ -218,6 +215,10 @@ int ref_is_automatic(const struct ref *ref) {\n \treturn ref->automatic;\n }\n \n+int ref_is_tag(const struct ref *ref) {\n+\treturn !prefixcmp(ref->name, \"refs/tags/\");\n+}\n+\n static struct ref *get_ref_map(struct transport *transport,\n \t\t\t       struct refspec *refs, int ref_count, int tags,\n \t\t\t       struct string_list *peeled_map,\n@@ -231,7 +232,7 @@ static struct ref *get_ref_map(struct transport *transport,\n \n \tconst struct ref *remote_refs = transport_get_remote_refs(transport);\n \n-\tif (ref_count || tags == TAGS_SET) {\n+\tif (ref_count) {\n \t\tfor (i = 0; i < ref_count; i++) {\n \t\t\tget_fetch_map(remote_refs, &refs[i], &tail, 0);\n \t\t\tif (refs[i].dst && refs[i].dst[0])\n@@ -240,8 +241,6 @@ static struct ref *get_ref_map(struct transport *transport,\n \t\t/* Merge everything on the command line, but not --tags */\n \t\tfor (rm = ref_map; rm; rm = rm->next)\n \t\t\trm->merge = 1;\n-\t\tif (tags == TAGS_SET)\n-\t\t\tget_fetch_map(remote_refs, tag_refspec, &tail, 0);\n \t} else {\n \t\t/* Use the defaults */\n \t\tstruct remote *remote = transport->remote;\n@@ -287,8 +286,24 @@ static struct ref *get_ref_map(struct transport *transport,\n \t\t}\n \t}\n \n-\tif (tags == TAGS_DEFAULT && *autotags)\n-\t\tadd_default_tags(remote_refs, &existing_refs, peeled_map, ref_map, &tail);\n+\tif (tags == TAGS_SET) {\n+\t\tadd_default_tags(remote_refs, &existing_refs, peeled_map, ref_map, 1, &tail);\n+\t\tdiscard_refs(&ref_map, 1, ref_is_tag);\n+\t\tfree_refs(rm);\n+\t} else if (tags == TAGS_DEFAULT && *autotags) {\n+\t\tadd_default_tags(remote_refs, &existing_refs, peeled_map, ref_map, 0, &tail);\n+\t\tfor (rm = ref_map; rm; rm = rm->next) {\n+\t\t\tstruct string_list_item *item;\n+\n+\t\t\tif (prefixcmp(rm->name, \"refs/tags/\"))\n+\t\t\t\tcontinue;\n+\t\t\titem = string_list_lookup(rm->name, peeled_map);\n+\t\t\trm->automatic = !(item && will_fetch(ref_map, item->util));\n+\t\t}\n+\t} else {\n+\t\tdiscard_refs(&ref_map, 0, ref_is_tag);\n+\t\tfree_refs(rm);\n+\t}\n \tstring_list_clear(&existing_refs, 0);\n \n \tref_remove_duplicates(ref_map);\n-- \n1.7.0.2.273.gc2413\n"},{"id":"137081","messageId":"cover.1268913163.git.agruen@suse.de","threadId":"23058","inReplyTo":null,"subject":"[PATCH 0/9] Multiple remotes without conflicts","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-18T11:52:43Z","receivedAt":"2010-03-18T11:52:43Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"Hello,\n\nI'm still trying to find a simple and painless way of sharing the object\nstore among multiple repositories: the idea is to have a \"parent\"\nrepository which contains the actual object store, and a number of\n\"child\" repositories which link to that object store.  The obvious\nproblem is garbage collection: we can only garbage collect the parent\nonce it has all refs of all its children.\n\nOne way of ensuring that is to make each child a \"remote\" of the parent,\nand to fetch all remotes first.  This works for branches, but not for\ntags or for the reflog.\n\nThe problem with tags is that they all share the same namespace (they\nare fetched with +refs/tags/*:refs/tags/*).  The obvious fix seems to be\nto use a different refspec for tags, but with the built-in default\nrefspec, tags are then fetched twice (for example, with\n+refs/*:refs/remotes/<remote>/*, refs/tags/foo is fetched as\nrefs/remotes/<remote>/tags/foo as well as refs/tags/foo).  Also, when\ntags are included in refspecs, automatic tag fetching and the --tags and\n--no-tags options break.\n\nThis patch series fixes this, and makes refspecs which match tags work\nin a reasonable way.  (There may be problems with pruning still; I did\nnot look into that, yet.)\n\nThe other issue is that the \"parent\" won't know about things\nreferenced in the child's reflog.  This is not an issue for the setup I\nhave in mind because the children won't have reflogs (they will only be\naccessed remotely), so I have not addressed this so far.\n\nHere is an example for setting up a shared object store:\n\n\tTOP=$PWD\n\tmkdir ab a b\n\n\tcd $TOP/ab\n\tgit init --bare\n\tfor x in a b; do\n\t\tgit remote add $x ../$x\n\t\tgit config --unset remote.$x.fetch\n\t\tgit config --add remote.$x.fetch \"+refs/*:refs/remotes/$x/*\"\n\tdone\n\n\tfor x in a b; do\n\t\tcd $TOP/$x\n\t\tgit init\n\t\tgit config gc.auto 0\n\t\trm -r .git/objects\n\t\tln -s ../../ab/objects .git/objects\n\tdone\n\nComments welcome!\n\nThanks,\nAndreas\n\n---\n\nAndreas Gruenbacher (9):\n  fetch: Check for a \"^{}\" suffix with suffixcmp()\n  fetch: Properly initialize refspec on stack\n  fetch: Fix minor memory leak\n  fetch: Move deepening fetch check into builtin/fetch.c\n  fetch: Move loop checking which refs we have already\n  fetch: Check if all objects exist after fetching\n  fetch: Use the same ref map for all branches and tags\n  fetch: Don't fetch tags twice\n  fetch: Make automatic tag following work with arbitrary refspecs\n\n builtin/fetch.c |  430 ++++++++++++++++++++++++++++++++-----------------------\n cache.h         |    4 +-\n transport.c     |   38 ++----\n 3 files changed, 264 insertions(+), 208 deletions(-)\n"},{"id":"137116","messageId":"20100318190712.GD10981@spearce.org","threadId":"23058","inReplyTo":"cover.1268913163.git.agruen@suse.de","subject":"Re: [PATCH 0/9] Multiple remotes without conflicts","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-18T19:07:12Z","receivedAt":"2010-03-18T19:07:12Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Andreas Gruenbacher <agruen@suse.de> wrote:\n> I'm still trying to find a simple and painless way of sharing the object\n> store among multiple repositories: the idea is to have a \"parent\"\n> repository which contains the actual object store, and a number of\n> \"child\" repositories which link to that object store.  The obvious\n> problem is garbage collection: we can only garbage collect the parent\n> once it has all refs of all its children.\n> \n> One way of ensuring that is to make each child a \"remote\" of the parent,\n> and to fetch all remotes first.  This works for branches, but not for\n> tags or for the reflog.\n\nThis just feels like the wrong solution.\n\nWhy can't we have a \"$GIT_DIR/children\" subdirectory with a symlink\nor file-containing-path to each child repository.  Modify the fsck\nand gc paths to include these additional reference and reflog spaces,\nand that's that.\n\nChild registration is then just a matter of installing the symlink\nin the parent, or removing it, and gc/fsck never needs to worry\nabout a fetch up front in order for it to be accurate.\n \n-- \nShawn.\n"},{"id":"137117","messageId":"20100318190816.GE10981@spearce.org","threadId":"23058","inReplyTo":"d3c00e2a27003dca196d5480007544610cc1e5b8.1268913163.git.agruen@suse.de","subject":"Re: [PATCH 6/9] fetch: Check if all objects exist after fetching","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-18T19:08:16Z","receivedAt":"2010-03-18T19:08:16Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Andreas Gruenbacher <agruen@suse.de> wrote:\n> Check if all objects reachable from the fetched refs exist after\n> fetching instead of before: this allows us to distinguish between a\n> repository which is not up to date and a corrupted repository, and to\n> ensure that the repository is up to date and complete after the fetch.\n\nI'm against this particular change because it looks like it breaks\nthe idea of \"quickfetch\", which we introduced to support faster\nfetches from the parent repository into a shared clone on the\nsame disk.\n \n-- \nShawn.\n"},{"id":"137124","messageId":"201003182036.46874.agruen@suse.de","threadId":"23058","inReplyTo":"20100318190816.GE10981@spearce.org","subject":"Re: [PATCH 6/9] fetch: Check if all objects exist after fetching","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-18T19:36:46Z","receivedAt":"2010-03-18T19:36:46Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"On Thursday 18 March 2010 20:08:16 Shawn O. Pearce wrote:\n> Andreas Gruenbacher <agruen@suse.de> wrote:\n> > Check if all objects reachable from the fetched refs exist after\n> > fetching instead of before: this allows us to distinguish between a\n> > repository which is not up to date and a corrupted repository, and to\n> > ensure that the repository is up to date and complete after the fetch.\n> \n> I'm against this particular change because it looks like it breaks\n> the idea of \"quickfetch\", which we introduced to support faster\n> fetches from the parent repository into a shared clone on the\n> same disk.\n\nI think you misunderstand the patch.  Before the patch, we were doing a rev-\nlist to determine if all objects needed are present.  If rev-list fails, this \ncan have two reasons: (a) some of the branches or tags needed do not exist, \n(b) all the branches and tags needed do exist, but other objects further up \nthe tree are missing (i.e., a corrupted repository).\n\nThe patch changes that to first check which needed objects are missing (with \nhas_sha1_file()), which is very efficient, by then fetching the objects which \nsurely need to be fetched, and by then checking the repository consistency \nwith rev-list.  If rev-list then fails, which should only happen in the rarest \ncases, we know that we need to fetch all branches and tags so that we are sure \nto catch missing objects further up the tree.\n\nSo we never fetch more than we did before, and in some cases, we fetch less. \nWe are also guaranteed to end up with a consistent repository in the end. (The \nold logic does not always guarantee that AFAICT: there seems to be one corner \ncase where a fetch succeeds without retrieving missing objects further up the \ntree.)\n\nThanks,\nAndreas\n"},{"id":"137127","messageId":"201003182042.49935.agruen@suse.de","threadId":"23058","inReplyTo":"20100318190712.GD10981@spearce.org","subject":"Re: [PATCH 0/9] Multiple remotes without conflicts","fromName":"Andreas Gruenbacher","fromEmail":"agruen@suse.de","sentAt":"2010-03-18T19:42:49Z","receivedAt":"2010-03-18T19:42:49Z","isPatch":true,"sender":{"key":"agruen@suse.de","avatar":null},"body":"On Thursday 18 March 2010 20:07:12 Shawn O. Pearce wrote:\n> Andreas Gruenbacher <agruen@suse.de> wrote:\n> > I'm still trying to find a simple and painless way of sharing the object\n> > store among multiple repositories: the idea is to have a \"parent\"\n> > repository which contains the actual object store, and a number of\n> > \"child\" repositories which link to that object store.  The obvious\n> > problem is garbage collection: we can only garbage collect the parent\n> > once it has all refs of all its children.\n> >\n> > One way of ensuring that is to make each child a \"remote\" of the parent,\n> > and to fetch all remotes first.  This works for branches, but not for\n> > tags or for the reflog.\n> \n> This just feels like the wrong solution.\n> \n> Why can't we have a \"$GIT_DIR/children\" subdirectory with a symlink\n> or file-containing-path to each child repository.  Modify the fsck\n> and gc paths to include these additional reference and reflog spaces,\n> and that's that.\n> \n> Child registration is then just a matter of installing the symlink\n> in the parent, or removing it, and gc/fsck never needs to worry\n> about a fetch up front in order for it to be accurate.\n\nAnother way would be to add some kind of \"child\" flag to remotes, and to teach \ngc to fetch such children before doing its work (or check that they are up-to-\ndate).  I think it's much nicer to construct the \"parent\" repository in such a \nway that it includes all relevant refs locally: it will just look and behave \nlike a \"normal\" repository.\n\nThanks,\nAndreas\n"}]}