{"thread":{"id":"48648","subject":"[PATCH 0/6] Refactor fetch negotiation into its own API","startedAt":"2018-06-04T17:29:45Z","lastAt":"2018-06-25T18:24:40Z","messageCount":47,"participants":["Jonathan Tan","Jonathan Nieder","Brandon Williams","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"349282","messageId":"cover.1527894919.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":null,"subject":"[PATCH 0/6] Refactor fetch negotiation into its own API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-04T17:29:30Z","receivedAt":"2018-06-04T17:29:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Both Stefan and Junio have suggested in [1] that I create a new\nnegotiation API first, then implement a new algorithm, and I perhaps too\noptimistically said that doing it all at once should be fine. Extracting\nthe API turns out to be more difficult than expected. Here are the\npatches that do that, without any new algorithms.\n\nThe patches include 3 patches that eliminate some minor differences\nbetween protocol v0 and protocol v2, then 3 patches that implement the\nAPI. I tried to write tests for all of the former 3, but could only do\nso for one, because I didn't know how to construct situations that show\na difference in behavior as an effect of such a code change (or, if such\na situation is possible without custom transports). As for the latter 3,\nthese are refactorings with no user-visible changes, and I have tried to\nwrite them in such a way that it is clear to a reviewer that the same\nfunctionality and logic is present before and after, and that they are\njust organized better.\n\nOverall, these do not change much functionality, but would be a good\nbase for contributors (myself and even others) to experiment with\nnegotiation algorithms.\n\n[1] https://public-inbox.org/git/20180521204340.260572-1-jonathantanmy@google.com/\n\nJonathan Tan (6):\n fetch-pack: clear marks before everything_local()\n fetch-pack: truly stop negotiation upon ACK ready\n fetch-pack: in protocol v2, enqueue commons first\n fetch-pack: make negotiation-related vars local\n fetch-pack: move common check and marking together\n fetch-pack: introduce negotiator API\n\n Makefile | 2 +\n fetch-negotiator.c | 7 ++\n fetch-negotiator.h | 45 +++++++++\n fetch-pack.c | 222 ++++++++++++------------------------------\n negotiator/default.c | 173 ++++++++++++++++++++++++++++++++\n negotiator/default.h | 8 ++\n object.h | 3 +-\n t/t5500-fetch-pack.sh | 35 +++++++\n 8 files changed, 332 insertions(+), 163 deletions(-)\n create mode 100644 fetch-negotiator.c\n create mode 100644 fetch-negotiator.h\n create mode 100644 negotiator/default.c\n create mode 100644 negotiator/default.h\n\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349283","messageId":"b9d6d8fef370fae316f78c851833dbd706ff6f7c.1527894919.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1527894919.git.jonathantanmy@google.com","subject":"[PATCH 1/6] fetch-pack: clear marks before everything_local()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-04T17:29:31Z","receivedAt":"2018-06-04T17:29:49Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"If tag following is required when using a transport that does not\nsupport tag following, fetch_pack() will be invoked twice in the same\nprocess, necessitating a clearing of the object flags used by\nfetch_pack() sometime during the second invocation. This is currently\ndone in find_common(), which means that the work done by\neverything_local() in marking complete remote refs as COMMON_REF is\nwasted.\n\nTo avoid this wastage, move this clearing from find_common() to its\nparent function do_fetch_pack(), right before it calls\neverything_local().\n\nThis has been occurring since the commit that introduced the clearing of\nmarks, 420e9af498 (\"Fix tag following\", 2008-03-19).\n\nThe corresponding code for protocol v2 in do_fetch_pack_v2() does not\nhave this problem, as the clearing of flags is done before any marking\n(whether by rev_list_insert_ref_oid() or everything_local()).\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex a320ce987..1358973a4 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -336,9 +336,6 @@ static int find_common(struct fetch_pack_args *args,\n \n \tif (args->stateless_rpc && multi_ack == 1)\n \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n-\tif (marked)\n-\t\tfor_each_ref(clear_marks, NULL);\n-\tmarked = 1;\n \n \tfor_each_ref(rev_list_insert_ref_oid, NULL);\n \tfor_each_cached_alternate(insert_one_alternate_object);\n@@ -1053,6 +1050,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n \n+\tif (marked)\n+\t\tfor_each_ref(clear_marks, NULL);\n+\tmarked = 1;\n \tif (everything_local(args, &ref, sought, nr_sought)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349284","messageId":"c79a6e6f96e2e453017669fabbed1a9fb4036cce.1527894919.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1527894919.git.jonathantanmy@google.com","subject":"[PATCH 2/6] fetch-pack: truly stop negotiation upon ACK ready","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-04T17:29:32Z","receivedAt":"2018-06-04T17:29:51Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"When \"ACK %s ready\" is received, find_common() clears rev_list in an\nattempt to stop further \"have\" lines from being sent [1]. This appears\nto work, despite the invocation to mark_common() in the \"while\" loop.\nThough it is possible for mark_common() to invoke rev_list_push() (thus\nmaking rev_list non-empty once more), it is more likely that the commits\nin rev_list that have un-SEEN parents are also unparsed, meaning that\nmark_common() is not invoked on them.\n\nTo avoid all this uncertainty, it is better to explicitly end the loop\nwhen \"ACK %s ready\" is received instead of clearing rev_list. Remove the\nclearing of rev_list and write \"if (got_ready) break;\" instead.\n\nThe corresponding code for protocol v2 in process_acks() does not have\nthe same problem, because the invoker of process_acks()\n(do_fetch_pack_v2()) proceeds immediately to processing the packfile\nupon \"ACK %s ready\". The clearing of rev_list here is thus redundant,\nand this patch also removes it.\n\n[1] The rationale is further described in the originating commit\nf2cba9299b (\"fetch-pack: Finish negotation if remote replies \"ACK %s\nready\"\", 2011-03-14).\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 1358973a4..2d090f612 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -517,10 +517,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\t\t\tmark_common(commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgot_continue = 1;\n-\t\t\t\t\tif (ack == ACK_ready) {\n-\t\t\t\t\t\tclear_prio_queue(&rev_list);\n+\t\t\t\t\tif (ack == ACK_ready)\n \t\t\t\t\t\tgot_ready = 1;\n-\t\t\t\t\t}\n \t\t\t\t\tbreak;\n \t\t\t\t\t}\n \t\t\t\t}\n@@ -530,6 +528,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\t\tprint_verbose(args, _(\"giving up\"));\n \t\t\t\tbreak; /* give up */\n \t\t\t}\n+\t\t\tif (got_ready)\n+\t\t\t\tbreak;\n \t\t}\n \t}\n done:\n@@ -1281,7 +1281,6 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n \t\t}\n \n \t\tif (!strcmp(reader->line, \"ready\")) {\n-\t\t\tclear_prio_queue(&rev_list);\n \t\t\treceived_ready = 1;\n \t\t\tcontinue;\n \t\t}\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349285","messageId":"8265d9485a2bfe0e81d68495afd3733879add992.1527894919.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1527894919.git.jonathantanmy@google.com","subject":"[PATCH 3/6] fetch-pack: in protocol v2, enqueue commons first","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-04T17:29:33Z","receivedAt":"2018-06-04T17:29:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In do_fetch_pack_v2(), rev_list_insert_ref_oid() is invoked before\neverything_local(). This means that if we have a commit that is both our\nref and their ref, it would be enqueued by rev_list_insert_ref_oid() as\nSEEN, and since it is thus already SEEN, everything_local() would not\nenqueue it.\n\nIf everything_local() were invoked first, as it is in do_fetch_pack()\nfor protocol v0, then everything_local() would enqueue it with\nCOMMON_REF | SEEN. The addition of COMMON_REF ensures that its parents\nare not sent as \"have\" lines.\n\nChange the order in do_fetch_pack_v2() to be consistent with\ndo_fetch_pack(), and to avoid sending unnecessary \"have\" lines.\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 6 +++---\n t/t5500-fetch-pack.sh | 35 +++++++++++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+), 3 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 2d090f612..192771a8f 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1372,14 +1372,14 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t\tfor_each_ref(clear_marks, NULL);\n \t\t\tmarked = 1;\n \n-\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n-\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tif (everything_local(args, &ref, sought, nr_sought))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n+\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n+\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n \t\t\tbreak;\n \t\tcase FETCH_SEND_REQUEST:\n \t\t\tif (send_fetch_request(fd[1], args, ref, &common,\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex 0680dec80..ad6a50ad6 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -808,6 +808,41 @@ test_expect_success 'fetch with --filter=blob:limit=0' '\n \tfetch_filter_blob_limit_zero server server\n '\n \n+test_expect_success 'use ref advertisement to prune \"have\" lines sent' '\n+\trm -rf server client &&\n+\tgit init server &&\n+\ttest_commit -C server aref_both_1 &&\n+\tgit -C server tag -d aref_both_1 &&\n+\ttest_commit -C server aref_both_2 &&\n+\n+\t# The ref name that only the server has must be a prefix of all the\n+\t# others, to ensure that the client has the same information regardless\n+\t# of whether protocol v0 (which does not have ref prefix filtering) or\n+\t# protocol v2 (which does) is used.\n+\tgit clone server client &&\n+\ttest_commit -C server aref &&\n+\ttest_commit -C client aref_client &&\n+\n+\t# In both protocol v0 and v2, ensure that the parent of aref_both_2 is\n+\t# not sent as a \"have\" line.\n+\n+\trm -f trace &&\n+\tcp -r client clientv0 &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv0 \\\n+\t\tfetch origin aref &&\n+\tgrep \"have $(git -C client rev-parse aref_client)\" trace &&\n+\tgrep \"have $(git -C client rev-parse aref_both_2)\" trace &&\n+\t! grep \"have $(git -C client rev-parse aref_both_2^)\" trace &&\n+\n+\trm -f trace &&\n+\tcp -r client clientv2 &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv2 -c protocol.version=2 \\\n+\t\tfetch origin aref &&\n+\tgrep \"have $(git -C client rev-parse aref_client)\" trace &&\n+\tgrep \"have $(git -C client rev-parse aref_both_2)\" trace &&\n+\t! grep \"have $(git -C client rev-parse aref_both_2^)\" trace\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349286","messageId":"8bda254e73fb00ff8448031002c7619bbe1dc285.1527894919.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1527894919.git.jonathantanmy@google.com","subject":"[PATCH 4/6] fetch-pack: make negotiation-related vars local","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-04T17:29:34Z","receivedAt":"2018-06-04T17:29:56Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Reduce the number of global variables by making the priority queue and\nthe count of non-common commits in it local, passing them as a struct to\nvarious functions where necessary.\n\nThis also helps in the case that fetch_pack() is invoked twice in the\nsame process (when tag following is required when using a transport that\ndoes not support tag following), in that different priority queues will\nnow be used in each invocation, instead of reusing the possibly\nnon-empty one.\n\nThe struct containing these variables is named \"data\" to ease review of\na subsequent patch in this series - in that patch, this struct\ndefinition and several functions will be moved to a negotiation-specific\nfile, and this allows the move to be verbatim.\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 104 +++++++++++++++++++++++++++++----------------------\n 1 file changed, 59 insertions(+), 45 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 192771a8f..ec92929bc 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -50,8 +50,12 @@ static int marked;\n */\n #define MAX_IN_VAIN 256\n \n-static struct prio_queue rev_list = { compare_commits_by_commit_date };\n-static int non_common_revs, multi_ack, use_sideband;\n+struct data {\n+\tstruct prio_queue rev_list;\n+\tint non_common_revs;\n+};\n+\n+static int multi_ack, use_sideband;\n /* Allow specifying sha1 if it is a ref tip. */\n #define ALLOW_TIP_SHA1\t01\n /* Allow request of a sha1 if it is reachable from a ref (possibly hidden ref). */\n@@ -93,7 +97,8 @@ static void cache_one_alternate(const char *refname,\n \tcache->items[cache->nr++] = obj;\n }\n \n-static void for_each_cached_alternate(void (*cb)(struct object *))\n+static void for_each_cached_alternate(struct data *data,\n+\t\t\t\t void (*cb)(struct data *, struct object *))\n {\n \tstatic int initialized;\n \tstatic struct alternate_object_cache cache;\n@@ -105,10 +110,10 @@ static void for_each_cached_alternate(void (*cb)(struct object *))\n \t}\n \n \tfor (i = 0; i < cache.nr; i++)\n-\t\tcb(cache.items[i]);\n+\t\tcb(data, cache.items[i]);\n }\n \n-static void rev_list_push(struct commit *commit, int mark)\n+static void rev_list_push(struct data *data, struct commit *commit, int mark)\n {\n \tif (!(commit->object.flags & mark)) {\n \t\tcommit->object.flags |= mark;\n@@ -116,19 +121,20 @@ static void rev_list_push(struct commit *commit, int mark)\n \t\tif (parse_commit(commit))\n \t\t\treturn;\n \n-\t\tprio_queue_put(&rev_list, commit);\n+\t\tprio_queue_put(&data->rev_list, commit);\n \n \t\tif (!(commit->object.flags & COMMON))\n-\t\t\tnon_common_revs++;\n+\t\t\tdata->non_common_revs++;\n \t}\n }\n \n-static int rev_list_insert_ref(const char *refname, const struct object_id *oid)\n+static int rev_list_insert_ref(struct data *data, const char *refname,\n+\t\t\t const struct object_id *oid)\n {\n \tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n \n \tif (o && o->type == OBJ_COMMIT)\n-\t\trev_list_push((struct commit *)o, SEEN);\n+\t\trev_list_push(data, (struct commit *)o, SEEN);\n \n \treturn 0;\n }\n@@ -136,7 +142,7 @@ static int rev_list_insert_ref(const char *refname, const struct object_id *oid)\n static int rev_list_insert_ref_oid(const char *refname, const struct object_id *oid,\n \t\t\t\t int flag, void *cb_data)\n {\n-\treturn rev_list_insert_ref(refname, oid);\n+\treturn rev_list_insert_ref(cb_data, refname, oid);\n }\n \n static int clear_marks(const char *refname, const struct object_id *oid,\n@@ -156,7 +162,7 @@ static int clear_marks(const char *refname, const struct object_id *oid,\n when only the server does not yet know that they are common).\n */\n \n-static void mark_common(struct commit *commit,\n+static void mark_common(struct data *data, struct commit *commit,\n \t\tint ancestors_only, int dont_parse)\n {\n \tif (commit != NULL && !(commit->object.flags & COMMON)) {\n@@ -166,12 +172,12 @@ static void mark_common(struct commit *commit,\n \t\t\to->flags |= COMMON;\n \n \t\tif (!(o->flags & SEEN))\n-\t\t\trev_list_push(commit, SEEN);\n+\t\t\trev_list_push(data, commit, SEEN);\n \t\telse {\n \t\t\tstruct commit_list *parents;\n \n \t\t\tif (!ancestors_only && !(o->flags & POPPED))\n-\t\t\t\tnon_common_revs--;\n+\t\t\t\tdata->non_common_revs--;\n \t\t\tif (!o->parsed && !dont_parse)\n \t\t\t\tif (parse_commit(commit))\n \t\t\t\t\treturn;\n@@ -179,7 +185,7 @@ static void mark_common(struct commit *commit,\n \t\t\tfor (parents = commit->parents;\n \t\t\t\t\tparents;\n \t\t\t\t\tparents = parents->next)\n-\t\t\t\tmark_common(parents->item, 0, dont_parse);\n+\t\t\t\tmark_common(data, parents->item, 0, dont_parse);\n \t\t}\n \t}\n }\n@@ -188,7 +194,7 @@ static void mark_common(struct commit *commit,\n Get the next rev to send, ignoring the common.\n */\n \n-static const struct object_id *get_rev(void)\n+static const struct object_id *get_rev(struct data *data)\n {\n \tstruct commit *commit = NULL;\n \n@@ -196,16 +202,16 @@ static const struct object_id *get_rev(void)\n \t\tunsigned int mark;\n \t\tstruct commit_list *parents;\n \n-\t\tif (rev_list.nr == 0 || non_common_revs == 0)\n+\t\tif (data->rev_list.nr == 0 || data->non_common_revs == 0)\n \t\t\treturn NULL;\n \n-\t\tcommit = prio_queue_get(&rev_list);\n+\t\tcommit = prio_queue_get(&data->rev_list);\n \t\tparse_commit(commit);\n \t\tparents = commit->parents;\n \n \t\tcommit->object.flags |= POPPED;\n \t\tif (!(commit->object.flags & COMMON))\n-\t\t\tnon_common_revs--;\n+\t\t\tdata->non_common_revs--;\n \n \t\tif (commit->object.flags & COMMON) {\n \t\t\t/* do not send \"have\", and ignore ancestors */\n@@ -220,9 +226,9 @@ static const struct object_id *get_rev(void)\n \n \t\twhile (parents) {\n \t\t\tif (!(parents->item->object.flags & SEEN))\n-\t\t\t\trev_list_push(parents->item, mark);\n+\t\t\t\trev_list_push(data, parents->item, mark);\n \t\t\tif (mark & COMMON)\n-\t\t\t\tmark_common(parents->item, 1, 0);\n+\t\t\t\tmark_common(data, parents->item, 1, 0);\n \t\t\tparents = parents->next;\n \t\t}\n \t}\n@@ -296,9 +302,9 @@ static void send_request(struct fetch_pack_args *args,\n \t\twrite_or_die(fd, buf->buf, buf->len);\n }\n \n-static void insert_one_alternate_object(struct object *obj)\n+static void insert_one_alternate_object(struct data *data, struct object *obj)\n {\n-\trev_list_insert_ref(NULL, &obj->oid);\n+\trev_list_insert_ref(data, NULL, &obj->oid);\n }\n \n #define INITIAL_FLUSH 16\n@@ -321,7 +327,7 @@ static int next_flush(int stateless_rpc, int count)\n \treturn count;\n }\n \n-static int find_common(struct fetch_pack_args *args,\n+static int find_common(struct data *data, struct fetch_pack_args *args,\n \t\t int fd[2], struct object_id *result_oid,\n \t\t struct ref *refs)\n {\n@@ -337,8 +343,8 @@ static int find_common(struct fetch_pack_args *args,\n \tif (args->stateless_rpc && multi_ack == 1)\n \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n \n-\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\tfor_each_cached_alternate(insert_one_alternate_object);\n+\tfor_each_ref(rev_list_insert_ref_oid, data);\n+\tfor_each_cached_alternate(data, insert_one_alternate_object);\n \n \tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\n@@ -456,7 +462,7 @@ static int find_common(struct fetch_pack_args *args,\n \tretval = -1;\n \tif (args->no_dependents)\n \t\tgoto done;\n-\twhile ((oid = get_rev())) {\n+\twhile ((oid = get_rev(data))) {\n \t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tprint_verbose(args, \"have %s\", oid_to_hex(oid));\n \t\tin_vain++;\n@@ -514,7 +520,7 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\t\t\t} else if (!args->stateless_rpc\n \t\t\t\t\t\t || ack != ACK_common)\n \t\t\t\t\t\tin_vain = 0;\n-\t\t\t\t\tmark_common(commit, 0, 1);\n+\t\t\t\t\tmark_common(data, commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgot_continue = 1;\n \t\t\t\t\tif (ack == ACK_ready)\n@@ -704,7 +710,7 @@ static void filter_refs(struct fetch_pack_args *args,\n \t*refs = newlist;\n }\n \n-static void mark_alternate_complete(struct object *obj)\n+static void mark_alternate_complete(struct data *unused, struct object *obj)\n {\n \tmark_complete(&obj->oid);\n }\n@@ -731,7 +737,8 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n \treturn 0;\n }\n \n-static int everything_local(struct fetch_pack_args *args,\n+static int everything_local(struct data *data,\n+\t\t\t struct fetch_pack_args *args,\n \t\t\t struct ref **refs,\n \t\t\t struct ref **sought, int nr_sought)\n {\n@@ -784,7 +791,7 @@ static int everything_local(struct fetch_pack_args *args,\n \tif (!args->no_dependents) {\n \t\tif (!args->deepen) {\n \t\t\tfor_each_ref(mark_complete_oid, NULL);\n-\t\t\tfor_each_cached_alternate(mark_alternate_complete);\n+\t\t\tfor_each_cached_alternate(NULL, mark_alternate_complete);\n \t\t\tcommit_list_sort_by_date(&complete);\n \t\t\tif (cutoff)\n \t\t\t\tmark_recent_complete_commits(args, cutoff);\n@@ -802,9 +809,10 @@ static int everything_local(struct fetch_pack_args *args,\n \t\t\t\tcontinue;\n \n \t\t\tif (!(o->flags & SEEN)) {\n-\t\t\t\trev_list_push((struct commit *)o, COMMON_REF | SEEN);\n+\t\t\t\trev_list_push(data, (struct commit *)o,\n+\t\t\t\t\t COMMON_REF | SEEN);\n \n-\t\t\t\tmark_common((struct commit *)o, 1, 1);\n+\t\t\t\tmark_common(data, (struct commit *)o, 1, 1);\n \t\t\t}\n \t\t}\n \t}\n@@ -978,6 +986,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tstruct object_id oid;\n \tconst char *agent_feature;\n \tint agent_len;\n+\tstruct data data = { { compare_commits_by_commit_date } };\n \n \tsort_ref_list(&ref, ref_compare_name);\n \tQSORT(sought, nr_sought, cmp_ref_by_name);\n@@ -1053,11 +1062,11 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (marked)\n \t\tfor_each_ref(clear_marks, NULL);\n \tmarked = 1;\n-\tif (everything_local(args, &ref, sought, nr_sought)) {\n+\tif (everything_local(&data, args, &ref, sought, nr_sought)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n-\tif (find_common(args, fd, &oid, ref) < 0)\n+\tif (find_common(&data, args, fd, &oid, ref) < 0)\n \t\tif (!args->keep_pack)\n \t\t\t/* When cloning, it is not unusual to have\n \t\t\t * no common commit.\n@@ -1138,13 +1147,14 @@ static void add_common(struct strbuf *req_buf, struct oidset *common)\n \t}\n }\n \n-static int add_haves(struct strbuf *req_buf, int *haves_to_send, int *in_vain)\n+static int add_haves(struct data *data, struct strbuf *req_buf,\n+\t\t int *haves_to_send, int *in_vain)\n {\n \tint ret = 0;\n \tint haves_added = 0;\n \tconst struct object_id *oid;\n \n-\twhile ((oid = get_rev())) {\n+\twhile ((oid = get_rev(data))) {\n \t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tif (++haves_added >= *haves_to_send)\n \t\t\tbreak;\n@@ -1163,7 +1173,8 @@ static int add_haves(struct strbuf *req_buf, int *haves_to_send, int *in_vain)\n \treturn ret;\n }\n \n-static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n+static int send_fetch_request(struct data *data, int fd_out,\n+\t\t\t const struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain)\n {\n@@ -1219,7 +1230,7 @@ static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n \t\tadd_common(&req_buf, common);\n \n \t\t/* Add initial haves */\n-\t\tret = add_haves(&req_buf, haves_to_send, in_vain);\n+\t\tret = add_haves(data, &req_buf, haves_to_send, in_vain);\n \t}\n \n \t/* Send request */\n@@ -1256,7 +1267,8 @@ static int process_section_header(struct packet_reader *reader,\n \treturn ret;\n }\n \n-static int process_acks(struct packet_reader *reader, struct oidset *common)\n+static int process_acks(struct data *data, struct packet_reader *reader,\n+\t\t\tstruct oidset *common)\n {\n \t/* received */\n \tint received_ready = 0;\n@@ -1275,7 +1287,7 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n \t\t\t\tstruct commit *commit;\n \t\t\t\toidset_insert(common, &oid);\n \t\t\t\tcommit = lookup_commit(&oid);\n-\t\t\t\tmark_common(commit, 0, 1);\n+\t\t\t\tmark_common(data, commit, 0, 1);\n \t\t\t}\n \t\t\tcontinue;\n \t\t}\n@@ -1353,6 +1365,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tstruct packet_reader reader;\n \tint in_vain = 0;\n \tint haves_to_send = INITIAL_FLUSH;\n+\tstruct data data = { { compare_commits_by_commit_date } };\n \tpacket_reader_init(&reader, fd[0], NULL, 0,\n \t\t\t PACKET_READ_CHOMP_NEWLINE);\n \n@@ -1373,16 +1386,17 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tmarked = 1;\n \n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n-\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n+\t\t\tif (everything_local(&data, args, &ref, sought, nr_sought))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n \n-\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, &data);\n+\t\t\tfor_each_cached_alternate(&data,\n+\t\t\t\t\t\t insert_one_alternate_object);\n \t\t\tbreak;\n \t\tcase FETCH_SEND_REQUEST:\n-\t\t\tif (send_fetch_request(fd[1], args, ref, &common,\n+\t\t\tif (send_fetch_request(&data, fd[1], args, ref, &common,\n \t\t\t\t\t &haves_to_send, &in_vain))\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\telse\n@@ -1390,7 +1404,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tbreak;\n \t\tcase FETCH_PROCESS_ACKS:\n \t\t\t/* Process ACKs/NAKs */\n-\t\t\tswitch (process_acks(&reader, &common)) {\n+\t\t\tswitch (process_acks(&data, &reader, &common)) {\n \t\t\tcase 2:\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\t\tbreak;\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349287","messageId":"02577402adaf0d109f74e59486d8a23f6ae68663.1527894919.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1527894919.git.jonathantanmy@google.com","subject":"[PATCH 5/6] fetch-pack: move common check and marking together","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-04T17:29:35Z","receivedAt":"2018-06-04T17:30:01Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This enables the calculation of was_common and the invocation to\nmark_common() to be abstracted into a single call to the negotiator API\n(to be introduced in a subsequent patch).\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex ec92929bc..54dd3feb8 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -499,11 +499,14 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \t\t\t\tcase ACK_continue: {\n \t\t\t\t\tstruct commit *commit =\n \t\t\t\t\t\tlookup_commit(result_oid);\n+\t\t\t\t\tint was_common;\n \t\t\t\t\tif (!commit)\n \t\t\t\t\t\tdie(_(\"invalid commit %s\"), oid_to_hex(result_oid));\n+\t\t\t\t\twas_common = commit->object.flags & COMMON;\n+\t\t\t\t\tmark_common(data, commit, 0, 1);\n \t\t\t\t\tif (args->stateless_rpc\n \t\t\t\t\t && ack == ACK_common\n-\t\t\t\t\t && !(commit->object.flags & COMMON)) {\n+\t\t\t\t\t && !was_common) {\n \t\t\t\t\t\t/* We need to replay the have for this object\n \t\t\t\t\t\t * on the next RPC request so the peer knows\n \t\t\t\t\t\t * it is in common with us.\n@@ -520,7 +523,6 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \t\t\t\t\t} else if (!args->stateless_rpc\n \t\t\t\t\t\t || ack != ACK_common)\n \t\t\t\t\t\tin_vain = 0;\n-\t\t\t\t\tmark_common(data, commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgot_continue = 1;\n \t\t\t\t\tif (ack == ACK_ready)\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349288","messageId":"c7e504a4664bb6008de970671af13eda7d85dc5b.1527894919.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1527894919.git.jonathantanmy@google.com","subject":"[PATCH 6/6] fetch-pack: introduce negotiator API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-04T17:29:36Z","receivedAt":"2018-06-04T17:30:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Introduce the new files fetch-negotiator.{h,c}, which contains an API\nbehind which the details of negotiation are abstracted. Currently, only\none algorithm is available: the existing one.\n\nThis patch is written to be more easily reviewed: static functions are\nmoved verbatim from fetch-pack.c to negotiator/default.c, and it can be\nseen that the lines replaced by negotiator->X() calls are present in the\nX() functions respectively.\n\nSigned-off-by: Jonathan Tan \n---\n Makefile | 2 +\n fetch-negotiator.c | 7 ++\n fetch-negotiator.h | 45 ++++++++++\n fetch-pack.c | 207 ++++++++++---------------------------------\n negotiator/default.c | 173 ++++++++++++++++++++++++++++++++++++\n negotiator/default.h | 8 ++\n object.h | 3 +-\n 7 files changed, 282 insertions(+), 163 deletions(-)\n create mode 100644 fetch-negotiator.c\n create mode 100644 fetch-negotiator.h\n create mode 100644 negotiator/default.c\n create mode 100644 negotiator/default.h\n\ndiff --git a/Makefile b/Makefile\nindex 4bca65383..8f1df08ac 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -859,6 +859,7 @@ LIB_OBJS += ewah/ewah_bitmap.o\n LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec-cmd.o\n+LIB_OBJS += fetch-negotiator.o\n LIB_OBJS += fetch-object.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n@@ -891,6 +892,7 @@ LIB_OBJS += merge-blobs.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += mergesort.o\n LIB_OBJS += name-hash.o\n+LIB_OBJS += negotiator/default.o\n LIB_OBJS += notes.o\n LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\ndiff --git a/fetch-negotiator.c b/fetch-negotiator.c\nnew file mode 100644\nindex 000000000..756c35d15\n--- /dev/null\n+++ b/fetch-negotiator.c\n@@ -0,0 +1,7 @@\n+#include \"fetch-negotiator.h\"\n+#include \"negotiator/default.h\"\n+\n+void fetch_negotiator_init(struct fetch_negotiator *negotiator)\n+{\n+\tdefault_negotiator_init(negotiator);\n+}\ndiff --git a/fetch-negotiator.h b/fetch-negotiator.h\nnew file mode 100644\nindex 000000000..b717b80dd\n--- /dev/null\n+++ b/fetch-negotiator.h\n@@ -0,0 +1,45 @@\n+#ifndef FETCH_NEGOTIATOR\n+#define FETCH_NEGOTIATOR\n+\n+#include \"cache.h\"\n+#include \"commit.h\"\n+\n+struct fetch_negotiator {\n+\t/*\n+\t * Before negotiation starts, indicate that the server is known to have\n+\t * this commit.\n+\t */\n+\tvoid (*known_common)(struct fetch_negotiator *, struct commit *);\n+\n+\t/*\n+\t * Once this function is invoked, known_common() cannot be invoked any\n+\t * more.\n+\t *\n+\t * Indicate that this commit and all its ancestors are to be checked\n+\t * for commonality with the server.\n+\t */\n+\tvoid (*add_tip)(struct fetch_negotiator *, struct commit *);\n+\n+\t/*\n+\t * Once this function is invoked, known_common() and add_tip() cannot\n+\t * be invoked any more.\n+\t *\n+\t * Return the next commit that the client should send as a \"have\" line.\n+\t */\n+\tconst struct object_id *(*next)(struct fetch_negotiator *);\n+\n+\t/*\n+\t * Inform the negotiator that the server has the given commit. This\n+\t * method must only be called on commits returned by next().\n+\t */\n+\tint (*ack)(struct fetch_negotiator *, struct commit *);\n+\n+\tvoid (*release)(struct fetch_negotiator *);\n+\n+\t/* internal use */\n+\tvoid *data;\n+};\n+\n+void fetch_negotiator_init(struct fetch_negotiator *negotiator);\n+\n+#endif\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 54dd3feb8..a399a62e1 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -15,10 +15,10 @@\n #include \"connect.h\"\n #include \"transport.h\"\n #include \"version.h\"\n-#include \"prio-queue.h\"\n #include \"sha1-array.h\"\n #include \"oidset.h\"\n #include \"packfile.h\"\n+#include \"fetch-negotiator.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -36,13 +36,7 @@ static const char *alternate_shallow_file;\n \n /* Remember to update object flag allocation in object.h */\n #define COMPLETE\t(1U << 0)\n-#define COMMON\t\t(1U << 1)\n-#define COMMON_REF\t(1U << 2)\n-#define SEEN\t\t(1U << 3)\n-#define POPPED\t\t(1U << 4)\n-#define ALTERNATE\t(1U << 5)\n-\n-static int marked;\n+#define ALTERNATE\t(1U << 1)\n \n /*\n * After sending this many \"have\"s if we do not get any new ACK , we\n@@ -50,11 +44,6 @@ static int marked;\n */\n #define MAX_IN_VAIN 256\n \n-struct data {\n-\tstruct prio_queue rev_list;\n-\tint non_common_revs;\n-};\n-\n static int multi_ack, use_sideband;\n /* Allow specifying sha1 if it is a ref tip. */\n #define ALLOW_TIP_SHA1\t01\n@@ -97,8 +86,8 @@ static void cache_one_alternate(const char *refname,\n \tcache->items[cache->nr++] = obj;\n }\n \n-static void for_each_cached_alternate(struct data *data,\n-\t\t\t\t void (*cb)(struct data *, struct object *))\n+static void for_each_cached_alternate(struct fetch_negotiator *negotiator,\n+\t\t\t\t void (*cb)(struct fetch_negotiator *, struct object *))\n {\n \tstatic int initialized;\n \tstatic struct alternate_object_cache cache;\n@@ -110,31 +99,17 @@ static void for_each_cached_alternate(struct data *data,\n \t}\n \n \tfor (i = 0; i < cache.nr; i++)\n-\t\tcb(data, cache.items[i]);\n-}\n-\n-static void rev_list_push(struct data *data, struct commit *commit, int mark)\n-{\n-\tif (!(commit->object.flags & mark)) {\n-\t\tcommit->object.flags |= mark;\n-\n-\t\tif (parse_commit(commit))\n-\t\t\treturn;\n-\n-\t\tprio_queue_put(&data->rev_list, commit);\n-\n-\t\tif (!(commit->object.flags & COMMON))\n-\t\t\tdata->non_common_revs++;\n-\t}\n+\t\tcb(negotiator, cache.items[i]);\n }\n \n-static int rev_list_insert_ref(struct data *data, const char *refname,\n+static int rev_list_insert_ref(struct fetch_negotiator *negotiator,\n+\t\t\t const char *refname,\n \t\t\t const struct object_id *oid)\n {\n \tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n \n \tif (o && o->type == OBJ_COMMIT)\n-\t\trev_list_push(data, (struct commit *)o, SEEN);\n+\t\tnegotiator->add_tip(negotiator, (struct commit *)o);\n \n \treturn 0;\n }\n@@ -145,97 +120,6 @@ static int rev_list_insert_ref_oid(const char *refname, const struct object_id *\n \treturn rev_list_insert_ref(cb_data, refname, oid);\n }\n \n-static int clear_marks(const char *refname, const struct object_id *oid,\n-\t\t int flag, void *cb_data)\n-{\n-\tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n-\n-\tif (o && o->type == OBJ_COMMIT)\n-\t\tclear_commit_marks((struct commit *)o,\n-\t\t\t\t COMMON | COMMON_REF | SEEN | POPPED);\n-\treturn 0;\n-}\n-\n-/*\n- This function marks a rev and its ancestors as common.\n- In some cases, it is desirable to mark only the ancestors (for example\n- when only the server does not yet know that they are common).\n-*/\n-\n-static void mark_common(struct data *data, struct commit *commit,\n-\t\tint ancestors_only, int dont_parse)\n-{\n-\tif (commit != NULL && !(commit->object.flags & COMMON)) {\n-\t\tstruct object *o = (struct object *)commit;\n-\n-\t\tif (!ancestors_only)\n-\t\t\to->flags |= COMMON;\n-\n-\t\tif (!(o->flags & SEEN))\n-\t\t\trev_list_push(data, commit, SEEN);\n-\t\telse {\n-\t\t\tstruct commit_list *parents;\n-\n-\t\t\tif (!ancestors_only && !(o->flags & POPPED))\n-\t\t\t\tdata->non_common_revs--;\n-\t\t\tif (!o->parsed && !dont_parse)\n-\t\t\t\tif (parse_commit(commit))\n-\t\t\t\t\treturn;\n-\n-\t\t\tfor (parents = commit->parents;\n-\t\t\t\t\tparents;\n-\t\t\t\t\tparents = parents->next)\n-\t\t\t\tmark_common(data, parents->item, 0, dont_parse);\n-\t\t}\n-\t}\n-}\n-\n-/*\n- Get the next rev to send, ignoring the common.\n-*/\n-\n-static const struct object_id *get_rev(struct data *data)\n-{\n-\tstruct commit *commit = NULL;\n-\n-\twhile (commit == NULL) {\n-\t\tunsigned int mark;\n-\t\tstruct commit_list *parents;\n-\n-\t\tif (data->rev_list.nr == 0 || data->non_common_revs == 0)\n-\t\t\treturn NULL;\n-\n-\t\tcommit = prio_queue_get(&data->rev_list);\n-\t\tparse_commit(commit);\n-\t\tparents = commit->parents;\n-\n-\t\tcommit->object.flags |= POPPED;\n-\t\tif (!(commit->object.flags & COMMON))\n-\t\t\tdata->non_common_revs--;\n-\n-\t\tif (commit->object.flags & COMMON) {\n-\t\t\t/* do not send \"have\", and ignore ancestors */\n-\t\t\tcommit = NULL;\n-\t\t\tmark = COMMON | SEEN;\n-\t\t} else if (commit->object.flags & COMMON_REF)\n-\t\t\t/* send \"have\", and ignore ancestors */\n-\t\t\tmark = COMMON | SEEN;\n-\t\telse\n-\t\t\t/* send \"have\", also for its ancestors */\n-\t\t\tmark = SEEN;\n-\n-\t\twhile (parents) {\n-\t\t\tif (!(parents->item->object.flags & SEEN))\n-\t\t\t\trev_list_push(data, parents->item, mark);\n-\t\t\tif (mark & COMMON)\n-\t\t\t\tmark_common(data, parents->item, 1, 0);\n-\t\t\tparents = parents->next;\n-\t\t}\n-\t}\n-\n-\treturn &commit->object.oid;\n-}\n-\n enum ack_type {\n \tNAK = 0,\n \tACK,\n@@ -302,9 +186,10 @@ static void send_request(struct fetch_pack_args *args,\n \t\twrite_or_die(fd, buf->buf, buf->len);\n }\n \n-static void insert_one_alternate_object(struct data *data, struct object *obj)\n+static void insert_one_alternate_object(struct fetch_negotiator *negotiator,\n+\t\t\t\t\tstruct object *obj)\n {\n-\trev_list_insert_ref(data, NULL, &obj->oid);\n+\trev_list_insert_ref(negotiator, NULL, &obj->oid);\n }\n \n #define INITIAL_FLUSH 16\n@@ -327,7 +212,8 @@ static int next_flush(int stateless_rpc, int count)\n \treturn count;\n }\n \n-static int find_common(struct data *data, struct fetch_pack_args *args,\n+static int find_common(struct fetch_negotiator *negotiator,\n+\t\t struct fetch_pack_args *args,\n \t\t int fd[2], struct object_id *result_oid,\n \t\t struct ref *refs)\n {\n@@ -343,8 +229,8 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \tif (args->stateless_rpc && multi_ack == 1)\n \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n \n-\tfor_each_ref(rev_list_insert_ref_oid, data);\n-\tfor_each_cached_alternate(data, insert_one_alternate_object);\n+\tfor_each_ref(rev_list_insert_ref_oid, negotiator);\n+\tfor_each_cached_alternate(negotiator, insert_one_alternate_object);\n \n \tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\n@@ -462,7 +348,7 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \tretval = -1;\n \tif (args->no_dependents)\n \t\tgoto done;\n-\twhile ((oid = get_rev(data))) {\n+\twhile ((oid = negotiator->next(negotiator))) {\n \t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tprint_verbose(args, \"have %s\", oid_to_hex(oid));\n \t\tin_vain++;\n@@ -502,8 +388,7 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \t\t\t\t\tint was_common;\n \t\t\t\t\tif (!commit)\n \t\t\t\t\t\tdie(_(\"invalid commit %s\"), oid_to_hex(result_oid));\n-\t\t\t\t\twas_common = commit->object.flags & COMMON;\n-\t\t\t\t\tmark_common(data, commit, 0, 1);\n+\t\t\t\t\twas_common = negotiator->ack(negotiator, commit);\n \t\t\t\t\tif (args->stateless_rpc\n \t\t\t\t\t && ack == ACK_common\n \t\t\t\t\t && !was_common) {\n@@ -712,7 +597,8 @@ static void filter_refs(struct fetch_pack_args *args,\n \t*refs = newlist;\n }\n \n-static void mark_alternate_complete(struct data *unused, struct object *obj)\n+static void mark_alternate_complete(struct fetch_negotiator *unused,\n+\t\t\t\t struct object *obj)\n {\n \tmark_complete(&obj->oid);\n }\n@@ -739,7 +625,7 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n \treturn 0;\n }\n \n-static int everything_local(struct data *data,\n+static int everything_local(struct fetch_negotiator *negotiator,\n \t\t\t struct fetch_pack_args *args,\n \t\t\t struct ref **refs,\n \t\t\t struct ref **sought, int nr_sought)\n@@ -810,12 +696,8 @@ static int everything_local(struct data *data,\n \t\t\tif (!o || o->type != OBJ_COMMIT || !(o->flags & COMPLETE))\n \t\t\t\tcontinue;\n \n-\t\t\tif (!(o->flags & SEEN)) {\n-\t\t\t\trev_list_push(data, (struct commit *)o,\n-\t\t\t\t\t COMMON_REF | SEEN);\n-\n-\t\t\t\tmark_common(data, (struct commit *)o, 1, 1);\n-\t\t\t}\n+\t\t\tnegotiator->known_common(negotiator,\n+\t\t\t\t\t\t (struct commit *)o);\n \t\t}\n \t}\n \n@@ -988,7 +870,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tstruct object_id oid;\n \tconst char *agent_feature;\n \tint agent_len;\n-\tstruct data data = { { compare_commits_by_commit_date } };\n+\tstruct fetch_negotiator negotiator;\n+\tfetch_negotiator_init(&negotiator);\n \n \tsort_ref_list(&ref, ref_compare_name);\n \tQSORT(sought, nr_sought, cmp_ref_by_name);\n@@ -1061,19 +944,17 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n \n-\tif (marked)\n-\t\tfor_each_ref(clear_marks, NULL);\n-\tmarked = 1;\n-\tif (everything_local(&data, args, &ref, sought, nr_sought)) {\n+\tif (everything_local(&negotiator, args, &ref, sought, nr_sought)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n-\tif (find_common(&data, args, fd, &oid, ref) < 0)\n+\tif (find_common(&negotiator, args, fd, &oid, ref) < 0)\n \t\tif (!args->keep_pack)\n \t\t\t/* When cloning, it is not unusual to have\n \t\t\t * no common commit.\n \t\t\t */\n \t\t\twarning(_(\"no common commits\"));\n+\tnegotiator.release(&negotiator);\n \n \tif (args->stateless_rpc)\n \t\tpacket_flush(fd[1]);\n@@ -1149,14 +1030,15 @@ static void add_common(struct strbuf *req_buf, struct oidset *common)\n \t}\n }\n \n-static int add_haves(struct data *data, struct strbuf *req_buf,\n+static int add_haves(struct fetch_negotiator *negotiator,\n+\t\t struct strbuf *req_buf,\n \t\t int *haves_to_send, int *in_vain)\n {\n \tint ret = 0;\n \tint haves_added = 0;\n \tconst struct object_id *oid;\n \n-\twhile ((oid = get_rev(data))) {\n+\twhile ((oid = negotiator->next(negotiator))) {\n \t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tif (++haves_added >= *haves_to_send)\n \t\t\tbreak;\n@@ -1175,7 +1057,7 @@ static int add_haves(struct data *data, struct strbuf *req_buf,\n \treturn ret;\n }\n \n-static int send_fetch_request(struct data *data, int fd_out,\n+static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t const struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain)\n@@ -1232,7 +1114,7 @@ static int send_fetch_request(struct data *data, int fd_out,\n \t\tadd_common(&req_buf, common);\n \n \t\t/* Add initial haves */\n-\t\tret = add_haves(data, &req_buf, haves_to_send, in_vain);\n+\t\tret = add_haves(negotiator, &req_buf, haves_to_send, in_vain);\n \t}\n \n \t/* Send request */\n@@ -1269,7 +1151,8 @@ static int process_section_header(struct packet_reader *reader,\n \treturn ret;\n }\n \n-static int process_acks(struct data *data, struct packet_reader *reader,\n+static int process_acks(struct fetch_negotiator *negotiator,\n+\t\t\tstruct packet_reader *reader,\n \t\t\tstruct oidset *common)\n {\n \t/* received */\n@@ -1289,7 +1172,7 @@ static int process_acks(struct data *data, struct packet_reader *reader,\n \t\t\t\tstruct commit *commit;\n \t\t\t\toidset_insert(common, &oid);\n \t\t\t\tcommit = lookup_commit(&oid);\n-\t\t\t\tmark_common(data, commit, 0, 1);\n+\t\t\t\tnegotiator->ack(negotiator, commit);\n \t\t\t}\n \t\t\tcontinue;\n \t\t}\n@@ -1367,7 +1250,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tstruct packet_reader reader;\n \tint in_vain = 0;\n \tint haves_to_send = INITIAL_FLUSH;\n-\tstruct data data = { { compare_commits_by_commit_date } };\n+\tstruct fetch_negotiator negotiator;\n+\tfetch_negotiator_init(&negotiator);\n \tpacket_reader_init(&reader, fd[0], NULL, 0,\n \t\t\t PACKET_READ_CHOMP_NEWLINE);\n \n@@ -1383,22 +1267,20 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n \t\t\t\targs->deepen = 1;\n \n-\t\t\tif (marked)\n-\t\t\t\tfor_each_ref(clear_marks, NULL);\n-\t\t\tmarked = 1;\n-\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n-\t\t\tif (everything_local(&data, args, &ref, sought, nr_sought))\n+\t\t\tif (everything_local(&negotiator, args, &ref, sought,\n+\t\t\t\t\t nr_sought))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n \n-\t\t\tfor_each_ref(rev_list_insert_ref_oid, &data);\n-\t\t\tfor_each_cached_alternate(&data,\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, &negotiator);\n+\t\t\tfor_each_cached_alternate(&negotiator,\n \t\t\t\t\t\t insert_one_alternate_object);\n \t\t\tbreak;\n \t\tcase FETCH_SEND_REQUEST:\n-\t\t\tif (send_fetch_request(&data, fd[1], args, ref, &common,\n+\t\t\tif (send_fetch_request(&negotiator, fd[1], args, ref,\n+\t\t\t\t\t &common,\n \t\t\t\t\t &haves_to_send, &in_vain))\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\telse\n@@ -1406,7 +1288,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tbreak;\n \t\tcase FETCH_PROCESS_ACKS:\n \t\t\t/* Process ACKs/NAKs */\n-\t\t\tswitch (process_acks(&data, &reader, &common)) {\n+\t\t\tswitch (process_acks(&negotiator, &reader, &common)) {\n \t\t\tcase 2:\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\t\tbreak;\n@@ -1434,6 +1316,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tcontinue;\n \t\t}\n \t}\n+\tnegotiator.release(&negotiator);\n \n \toidset_clear(&common);\n \treturn ref;\ndiff --git a/negotiator/default.c b/negotiator/default.c\nnew file mode 100644\nindex 000000000..48d290e06\n--- /dev/null\n+++ b/negotiator/default.c\n@@ -0,0 +1,173 @@\n+#include \"default.h\"\n+#include \"../prio-queue.h\"\n+#include \"../refs.h\"\n+#include \"../tag.h\"\n+\n+/* Remember to update object flag allocation in object.h */\n+#define COMMON\t\t(1U << 2)\n+#define COMMON_REF\t(1U << 3)\n+#define SEEN\t\t(1U << 4)\n+#define POPPED\t\t(1U << 5)\n+\n+static int marked;\n+\n+struct data {\n+\tstruct prio_queue rev_list;\n+\tint non_common_revs;\n+};\n+\n+static void rev_list_push(struct data *data, struct commit *commit, int mark)\n+{\n+\tif (!(commit->object.flags & mark)) {\n+\t\tcommit->object.flags |= mark;\n+\n+\t\tif (parse_commit(commit))\n+\t\t\treturn;\n+\n+\t\tprio_queue_put(&data->rev_list, commit);\n+\n+\t\tif (!(commit->object.flags & COMMON))\n+\t\t\tdata->non_common_revs++;\n+\t}\n+}\n+\n+static int clear_marks(const char *refname, const struct object_id *oid,\n+\t\t int flag, void *cb_data)\n+{\n+\tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n+\n+\tif (o && o->type == OBJ_COMMIT)\n+\t\tclear_commit_marks((struct commit *)o,\n+\t\t\t\t COMMON | COMMON_REF | SEEN | POPPED);\n+\treturn 0;\n+}\n+\n+/*\n+ This function marks a rev and its ancestors as common.\n+ In some cases, it is desirable to mark only the ancestors (for example\n+ when only the server does not yet know that they are common).\n+*/\n+\n+static void mark_common(struct data *data, struct commit *commit,\n+\t\tint ancestors_only, int dont_parse)\n+{\n+\tif (commit != NULL && !(commit->object.flags & COMMON)) {\n+\t\tstruct object *o = (struct object *)commit;\n+\n+\t\tif (!ancestors_only)\n+\t\t\to->flags |= COMMON;\n+\n+\t\tif (!(o->flags & SEEN))\n+\t\t\trev_list_push(data, commit, SEEN);\n+\t\telse {\n+\t\t\tstruct commit_list *parents;\n+\n+\t\t\tif (!ancestors_only && !(o->flags & POPPED))\n+\t\t\t\tdata->non_common_revs--;\n+\t\t\tif (!o->parsed && !dont_parse)\n+\t\t\t\tif (parse_commit(commit))\n+\t\t\t\t\treturn;\n+\n+\t\t\tfor (parents = commit->parents;\n+\t\t\t\t\tparents;\n+\t\t\t\t\tparents = parents->next)\n+\t\t\t\tmark_common(data, parents->item, 0, dont_parse);\n+\t\t}\n+\t}\n+}\n+\n+/*\n+ Get the next rev to send, ignoring the common.\n+*/\n+\n+static const struct object_id *get_rev(struct data *data)\n+{\n+\tstruct commit *commit = NULL;\n+\n+\twhile (commit == NULL) {\n+\t\tunsigned int mark;\n+\t\tstruct commit_list *parents;\n+\n+\t\tif (data->rev_list.nr == 0 || data->non_common_revs == 0)\n+\t\t\treturn NULL;\n+\n+\t\tcommit = prio_queue_get(&data->rev_list);\n+\t\tparse_commit(commit);\n+\t\tparents = commit->parents;\n+\n+\t\tcommit->object.flags |= POPPED;\n+\t\tif (!(commit->object.flags & COMMON))\n+\t\t\tdata->non_common_revs--;\n+\n+\t\tif (commit->object.flags & COMMON) {\n+\t\t\t/* do not send \"have\", and ignore ancestors */\n+\t\t\tcommit = NULL;\n+\t\t\tmark = COMMON | SEEN;\n+\t\t} else if (commit->object.flags & COMMON_REF)\n+\t\t\t/* send \"have\", and ignore ancestors */\n+\t\t\tmark = COMMON | SEEN;\n+\t\telse\n+\t\t\t/* send \"have\", also for its ancestors */\n+\t\t\tmark = SEEN;\n+\n+\t\twhile (parents) {\n+\t\t\tif (!(parents->item->object.flags & SEEN))\n+\t\t\t\trev_list_push(data, parents->item, mark);\n+\t\t\tif (mark & COMMON)\n+\t\t\t\tmark_common(data, parents->item, 1, 0);\n+\t\t\tparents = parents->next;\n+\t\t}\n+\t}\n+\n+\treturn &commit->object.oid;\n+}\n+\n+static void known_common(struct fetch_negotiator *n, struct commit *c)\n+{\n+\tif (!(c->object.flags & SEEN)) {\n+\t\trev_list_push(n->data, c, COMMON_REF | SEEN);\n+\t\tmark_common(n->data, c, 1, 1);\n+\t}\n+}\n+\n+static void add_tip(struct fetch_negotiator *n, struct commit *c)\n+{\n+\tn->known_common = NULL;\n+\trev_list_push(n->data, c, SEEN);\n+}\n+\n+static const struct object_id *next(struct fetch_negotiator *n)\n+{\n+\tn->known_common = NULL;\n+\tn->add_tip = NULL;\n+\treturn get_rev(n->data);\n+}\n+\n+static int ack(struct fetch_negotiator *n, struct commit *c)\n+{\n+\tint known_to_be_common = !!(c->object.flags & COMMON);\n+\tmark_common(n->data, c, 0, 1);\n+\treturn known_to_be_common;\n+}\n+\n+static void release(struct fetch_negotiator *n)\n+{\n+\tclear_prio_queue(&((struct data *)n->data)->rev_list);\n+\tFREE_AND_NULL(n->data);\n+}\n+\n+void default_negotiator_init(struct fetch_negotiator *negotiator)\n+{\n+\tstruct data *data;\n+\tnegotiator->known_common = known_common;\n+\tnegotiator->add_tip = add_tip;\n+\tnegotiator->next = next;\n+\tnegotiator->ack = ack;\n+\tnegotiator->release = release;\n+\tnegotiator->data = data = xcalloc(1, sizeof(*data));\n+\tdata->rev_list.compare = compare_commits_by_commit_date;\n+\n+\tif (marked)\n+\t\tfor_each_ref(clear_marks, NULL);\n+\tmarked = 1;\n+}\ndiff --git a/negotiator/default.h b/negotiator/default.h\nnew file mode 100644\nindex 000000000..6c4b7e526\n--- /dev/null\n+++ b/negotiator/default.h\n@@ -0,0 +1,8 @@\n+#ifndef NEGOTIATOR_DEFAULT_H\n+#define NEGOTIATOR_DEFAULT_H\n+\n+#include \"fetch-negotiator.h\"\n+\n+void default_negotiator_init(struct fetch_negotiator *negotiator);\n+\n+#endif\ndiff --git a/object.h b/object.h\nindex 5c1395500..7db4941d6 100644\n--- a/object.h\n+++ b/object.h\n@@ -28,7 +28,8 @@ struct object_array {\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\n- * fetch-pack.c: 0----5\n+ * fetch-pack.c: 01\n+ * negotiator/default.c: 2--5\n * walker.c: 0-2\n * upload-pack.c: 4 11----------------19\n * builtin/blame.c: 12-13\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349419","messageId":"20180605230821.GC9266@aiede.svl.corp.google.com","threadId":"48648","inReplyTo":"b9d6d8fef370fae316f78c851833dbd706ff6f7c.1527894919.git.jonathantanmy@google.com","subject":"Re: [PATCH 1/6] fetch-pack: clear marks before everything_local()","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-06-05T23:08:21Z","receivedAt":"2018-06-05T23:08:27Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJonathan Tan wrote:\n\n> If tag following is required when using a transport that does not\n> support tag following, fetch_pack() will be invoked twice in the same\n> process, necessitating a clearing of the object flags used by\n> fetch_pack() sometime during the second invocation. This is currently\n> done in find_common(), which means that the work done by\n> everything_local() in marking complete remote refs as COMMON_REF is\n> wasted.\n>\n> To avoid this wastage, move this clearing from find_common() to its\n> parent function do_fetch_pack(), right before it calls\n> everything_local().\n\nI had to read this a few times and didn't end up understanding it.\n\nIs the idea that this will speed something up? Can you provide e.g.\n\"perf stat\" output (or even a new perf test in t/perf) demonstrating\nthe improvement? Or is it a cleanup?\n\n[...]\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -336,9 +336,6 @@ static int find_common(struct fetch_pack_args *args,\n> \n> \tif (args->stateless_rpc && multi_ack == 1)\n> \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n> -\tif (marked)\n> -\t\tfor_each_ref(clear_marks, NULL);\n> -\tmarked = 1;\n> \n> \tfor_each_ref(rev_list_insert_ref_oid, NULL);\n> \tfor_each_cached_alternate(insert_one_alternate_object);\n> @@ -1053,6 +1050,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n> \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n> \t\tdie(_(\"Server does not support --deepen\"));\n> \n> +\tif (marked)\n> +\t\tfor_each_ref(clear_marks, NULL);\n> +\tmarked = 1;\n> \tif (everything_local(args, &ref, sought, nr_sought)) {\n\nAs an experiment, I tried applying the '-' part of the change without\nthe '+' part to get confidence that tests cover this well. To my\nchagrin, all tests still passed. :/\n\nIn the preimage, we call clear_marks in find_common. This is right\nbefore we start setting up the revision walk, e.g. by inserting\nrevisions for each ref. In the postimage, we call clear_marks in\ndo_fetch_pack. This is right before we call everything_local.\n\nI end up feeling that I don't understand the code well, neither before\nnor after the patch. Ideas for making it clearer?\n\nThanks,\nJonathan\n"},{"id":"349420","messageId":"20180605231634.GD9266@aiede.svl.corp.google.com","threadId":"48648","inReplyTo":"c79a6e6f96e2e453017669fabbed1a9fb4036cce.1527894919.git.jonathantanmy@google.com","subject":"Re: [PATCH 2/6] fetch-pack: truly stop negotiation upon ACK ready","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-06-05T23:16:34Z","receivedAt":"2018-06-05T23:16:40Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJonathan Tan wrote:\n\n> When \"ACK %s ready\" is received, find_common() clears rev_list in an\n> attempt to stop further \"have\" lines from being sent [1]. This appears\n> to work, despite the invocation to mark_common() in the \"while\" loop.\n\nDoes \"appears to work\" mean \"works\" or \"doesn't work but does an okay\njob of faking\"?\n\n> Though it is possible for mark_common() to invoke rev_list_push() (thus\n> making rev_list non-empty once more), it is more likely that the commits\n\nnit: s/more likely/most likely/\nor s/it is more likely that/usually/\n\n> in rev_list that have un-SEEN parents are also unparsed, meaning that\n> mark_common() is not invoked on them.\n>\n> To avoid all this uncertainty, it is better to explicitly end the loop\n> when \"ACK %s ready\" is received instead of clearing rev_list. Remove the\n> clearing of rev_list and write \"if (got_ready) break;\" instead.\n\nI'm still a little curious about whether this can happen in practice\nor whether it's just about readability (or whether you didn't figure\nout which, which is perfectly fine), but either way it's a good\nchange.\n\n> The corresponding code for protocol v2 in process_acks() does not have\n> the same problem, because the invoker of process_acks()\n> (do_fetch_pack_v2()) proceeds immediately to processing the packfile\n\nnit: s/proceeds/procedes/\n\n> upon \"ACK %s ready\". The clearing of rev_list here is thus redundant,\n> and this patch also removes it.\n>\n> [1] The rationale is further described in the originating commit\n> f2cba9299b (\"fetch-pack: Finish negotation if remote replies \"ACK %s\n> ready\"\", 2011-03-14).\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> fetch-pack.c | 7 +++----\n> 1 file changed, 3 insertions(+), 4 deletions(-)\n[...]\n> +++ b/fetch-pack.c\n> @@ -517,10 +517,8 @@ static int find_common(struct fetch_pack_args *args,\n> \t\t\t\t\tmark_common(commit, 0, 1);\n> \t\t\t\t\tretval = 0;\n> \t\t\t\t\tgot_continue = 1;\n> -\t\t\t\t\tif (ack == ACK_ready) {\n> -\t\t\t\t\t\tclear_prio_queue(&rev_list);\n> +\t\t\t\t\tif (ack == ACK_ready)\n> \t\t\t\t\t\tgot_ready = 1;\n> -\t\t\t\t\t}\n> \t\t\t\t\tbreak;\n> \t\t\t\t\t}\n> \t\t\t\t}\n> @@ -530,6 +528,8 @@ static int find_common(struct fetch_pack_args *args,\n> \t\t\t\tprint_verbose(args, _(\"giving up\"));\n> \t\t\t\tbreak; /* give up */\n> \t\t\t}\n> +\t\t\tif (got_ready)\n> +\t\t\t\tbreak;\n\nMakes sense.\n\n> @@ -1281,7 +1281,6 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n> \t\t}\n> \n> \t\tif (!strcmp(reader->line, \"ready\")) {\n> -\t\t\tclear_prio_queue(&rev_list);\n> \t\t\treceived_ready = 1;\n> \t\t\tcontinue;\n\nI'm curious about the lifetime of &rev_list. Does the priority queue\nget freed eventually?\n\nThanks,\nJonathan\n"},{"id":"349421","messageId":"20180605231829.GE9266@aiede.svl.corp.google.com","threadId":"48648","inReplyTo":"20180605231634.GD9266@aiede.svl.corp.google.com","subject":"Re: [PATCH 2/6] fetch-pack: truly stop negotiation upon ACK ready","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-06-05T23:18:29Z","receivedAt":"2018-06-05T23:18:35Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n> Jonathan Tan wrote:\n\n>> The corresponding code for protocol v2 in process_acks() does not have\n>> the same problem, because the invoker of process_acks()\n>> (do_fetch_pack_v2()) proceeds immediately to processing the packfile\n>\n> nit: s/proceeds/procedes/\n\nWhoops. My spellchecker deceived me.\n\nI even checked Wiktionary and found that it was a verb there and didn't\nbother to look at the definition:\n\n\tMisspelling of proceed\n\nYou already had the right spelling.\n\nSorry for the noise,\nJonathan\n"},{"id":"349422","messageId":"20180605233026.GF9266@aiede.svl.corp.google.com","threadId":"48648","inReplyTo":"8265d9485a2bfe0e81d68495afd3733879add992.1527894919.git.jonathantanmy@google.com","subject":"Re: [PATCH 3/6] fetch-pack: in protocol v2, enqueue commons first","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-06-05T23:30:26Z","receivedAt":"2018-06-05T23:30:32Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJonathan Tan wrote:\n\n> In do_fetch_pack_v2(), rev_list_insert_ref_oid() is invoked before\n> everything_local(). This means that if we have a commit that is both our\n> ref and their ref, it would be enqueued by rev_list_insert_ref_oid() as\n> SEEN, and since it is thus already SEEN, everything_local() would not\n> enqueue it.\n>\n> If everything_local() were invoked first, as it is in do_fetch_pack()\n> for protocol v0, then everything_local() would enqueue it with\n> COMMON_REF | SEEN. The addition of COMMON_REF ensures that its parents\n> are not sent as \"have\" lines.\n>\n> Change the order in do_fetch_pack_v2() to be consistent with\n> do_fetch_pack(), and to avoid sending unnecessary \"have\" lines.\n\nI get lost in the above description. I suspect it's doing a good job\nof describing the code, instead of answering the question I really\nhave about what is broken and what behavior we want instead.\n\nE.g. are there some commands that I can run to trigger the unnecessary\n\"have\" lines? That would make it easier for me to understand the rest\nand whether this is a good approach for suppressing them.\n\nIt's possible I should be skipping to the test, but a summary in the\ncommit message would make life easier for lazy people like me. :)\n\n[...]\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -1372,14 +1372,14 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\t\tfor_each_ref(clear_marks, NULL);\n> \t\t\tmarked = 1;\n> \n> -\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n> -\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n> -\n> \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> \t\t\tif (everything_local(args, &ref, sought, nr_sought))\n> \t\t\t\tstate = FETCH_DONE;\n> \t\t\telse\n> \t\t\t\tstate = FETCH_SEND_REQUEST;\n> +\n> +\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n> +\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n\nThis is subtle. My instinct would be to assume that the purpose of\neverything_local is just to determine whether we need to send a fetch\nrequest, but it appears we also want to rely on a side effect from it.\n\nCould everything_local get a function comment to describe what side\neffects we will be counting on from it?\n\n> \t\t\tbreak;\n> \t\tcase FETCH_SEND_REQUEST:\n> \t\t\tif (send_fetch_request(fd[1], args, ref, &common,\n> diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\n> index 0680dec80..ad6a50ad6 100755\n> --- a/t/t5500-fetch-pack.sh\n> +++ b/t/t5500-fetch-pack.sh\n> @@ -808,6 +808,41 @@ test_expect_success 'fetch with --filter=blob:limit=0' '\n> \tfetch_filter_blob_limit_zero server server\n> '\n> \n> +test_expect_success 'use ref advertisement to prune \"have\" lines sent' '\n\nnit: this adds the new test as last in the script. Is there some\nlogical earlier place in the file it can go instead? That way, the\nfile stays organized and concurrent patches that modify the same test\nscript are less likely to conflict.\n\n> +\trm -rf server client &&\n> +\tgit init server &&\n> +\ttest_commit -C server aref_both_1 &&\n> +\tgit -C server tag -d aref_both_1 &&\n> +\ttest_commit -C server aref_both_2 &&\n\nWhat does aref stand for?\n\n> +\n> +\t# The ref name that only the server has must be a prefix of all the\n> +\t# others, to ensure that the client has the same information regardless\n> +\t# of whether protocol v0 (which does not have ref prefix filtering) or\n> +\t# protocol v2 (which does) is used.\n\nmust or else what? Maybe:\n\n\t# In this test, the ref name that only the server has is a prefix of\n\t# all other refs. This ensures that the client has the same information\n\t# regardless of [etc]\n\n> +\tgit clone server client &&\n> +\ttest_commit -C server aref &&\n> +\ttest_commit -C client aref_client &&\n> +\n> +\t# In both protocol v0 and v2, ensure that the parent of aref_both_2 is\n> +\t# not sent as a \"have\" line.\n\nWhy shouldn't it be sent as a \"have\" line? E.g. does another \"have\"\nline make it redundant?\n\n> +\n> +\trm -f trace &&\n> +\tcp -r client clientv0 &&\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv0 \\\n> +\t\tfetch origin aref &&\n> +\tgrep \"have $(git -C client rev-parse aref_client)\" trace &&\n> +\tgrep \"have $(git -C client rev-parse aref_both_2)\" trace &&\n\nnit: can make this more robust by doing\n\n\taref_client=$(git -C client rev-parse aref_client) &&\n\taref_both_2=$(git -C client rev-parse aref_both_2) &&\n\nsince this way if the git command fails, the test fails.\n\n> +\t! grep \"have $(git -C client rev-parse aref_both_2^)\" trace &&\n\nNice.\n\nThanks for a pleasant read,\nJonathan\n"},{"id":"349423","messageId":"20180605233504.GG9266@aiede.svl.corp.google.com","threadId":"48648","inReplyTo":"8bda254e73fb00ff8448031002c7619bbe1dc285.1527894919.git.jonathantanmy@google.com","subject":"Re: [PATCH 4/6] fetch-pack: make negotiation-related vars local","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-06-05T23:35:04Z","receivedAt":"2018-06-05T23:35:10Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Tan wrote:\n\n> Reduce the number of global variables by making the priority queue and\n> the count of non-common commits in it local, passing them as a struct to\n> various functions where necessary.\n\n\\o/\n\n> This also helps in the case that fetch_pack() is invoked twice in the\n> same process (when tag following is required when using a transport that\n> does not support tag following), in that different priority queues will\n> now be used in each invocation, instead of reusing the possibly\n> non-empty one.\n>\n> The struct containing these variables is named \"data\" to ease review of\n> a subsequent patch in this series - in that patch, this struct\n> definition and several functions will be moved to a negotiation-specific\n> file, and this allows the move to be verbatim.\n\nHm. Is the idea that 'struct data' gets stored in the opaque 'data'\nmember of the fetch_negotiator?\n\n'struct data' is a quite vague type name --- it's almost equivalent to\n'void' (which I suppose is the idea). How about something like\n'struct negotiation_data' or 'fetch_negotiator_data' in this patch?\nThat way this last paragraph of the commit message wouldn't be needed.\n\n[...]\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -50,8 +50,12 @@ static int marked;\n> */\n> #define MAX_IN_VAIN 256\n> \n> -static struct prio_queue rev_list = { compare_commits_by_commit_date };\n> -static int non_common_revs, multi_ack, use_sideband;\n> +struct data {\n> +\tstruct prio_queue rev_list;\n> +\tint non_common_revs;\n> +};\n\nHow does this struct get used? What does it represent? A comment\nmight help.\n\nThe rest looks good.\n\nThanks,\nJonathan\n"},{"id":"349424","messageId":"20180606000146.GH9266@aiede.svl.corp.google.com","threadId":"48648","inReplyTo":"02577402adaf0d109f74e59486d8a23f6ae68663.1527894919.git.jonathantanmy@google.com","subject":"Re: [PATCH 5/6] fetch-pack: move common check and marking together","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-06-06T00:01:46Z","receivedAt":"2018-06-06T00:01:52Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJonathan Tan wrote:\n\n> This enables the calculation of was_common and the invocation to\n> mark_common() to be abstracted into a single call to the negotiator API\n> (to be introduced in a subsequent patch).\n\nI like it. I think it should be possible to describe the benefit of\nthis patch without reference to the specifics of the subsequent one.\nMaybe something like:\n\n\tWhen receiving 'ACK continue' for a common commit,\n\tcheck if the commit was already known to be common and mark it\n\tas such if not up front. This should make future refactoring\n\tof how the information about common commits is stored more\n\tstraightforward.\n\n\tNo visible change intended.\n\n> Signed-off-by: Jonathan Tan \n> ---\n> fetch-pack.c | 6 ++++--\n> 1 file changed, 4 insertions(+), 2 deletions(-)\n\nWith or without such a clarification,\nReviewed-by: Jonathan Nieder \n\nThanks.\n"},{"id":"349425","messageId":"20180605173214.c3920e67cbbbd463be33bc70@google.com","threadId":"48648","inReplyTo":"20180605230821.GC9266@aiede.svl.corp.google.com","subject":"Re: [PATCH 1/6] fetch-pack: clear marks before everything_local()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T00:32:14Z","receivedAt":"2018-06-06T00:32:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 5 Jun 2018 16:08:21 -0700\nJonathan Nieder wrote:\n\n> Hi,\n> \n> Jonathan Tan wrote:\n> \n> > If tag following is required when using a transport that does not\n> > support tag following, fetch_pack() will be invoked twice in the same\n> > process, necessitating a clearing of the object flags used by\n> > fetch_pack() sometime during the second invocation. This is currently\n> > done in find_common(), which means that the work done by\n> > everything_local() in marking complete remote refs as COMMON_REF is\n> > wasted.\n> >\n> > To avoid this wastage, move this clearing from find_common() to its\n> > parent function do_fetch_pack(), right before it calls\n> > everything_local().\n> \n> I had to read this a few times and didn't end up understanding it.\n> \n> Is the idea that this will speed something up? Can you provide e.g.\n> \"perf stat\" output (or even a new perf test in t/perf) demonstrating\n> the improvement? Or is it a cleanup?\n\nFirstly, I don't know of a practical way to demonstrate this, because we\ndon't have an implementation of a transport that does not support tag\nfollowing. If we could demonstrate this, I think we can demonstrating it\nby constructing a negotiation scenario in which COMMON_REF would have\nbeen helpful, e.g. the following (untested) scenario:\n\n T C (T=tag, C=commit)\n |/\n O\n\nWe run \"git fetch C\" and on the second round (when we fetch T), if we\nwiped the flags *after* everything_local() (as is currently done),\nnegotiation would send \"have C\" and \"have O\". But if we wiped the flags\n*before* everything_local(), then C would have the COMMON_REF flag and\nwe will see that we only send \"have C\". So we have more efficient\nnegotiation.\n\n> As an experiment, I tried applying the '-' part of the change without\n> the '+' part to get confidence that tests cover this well. To my\n> chagrin, all tests still passed. :/\n\nYes, because we don't have tests against a transport which doesn't\nsupport tag following.\n\n> In the preimage, we call clear_marks in find_common. This is right\n> before we start setting up the revision walk, e.g. by inserting\n> revisions for each ref. In the postimage, we call clear_marks in\n> do_fetch_pack. This is right before we call everything_local.\n> \n> I end up feeling that I don't understand the code well, neither before\n> nor after the patch. Ideas for making it clearer?\n\nOne idea is to first separate everything_local() into its side effect\nparts and the \"true\" check that everything is local. I'll do that and\nsend it as part of v2 of this patch series.\n"},{"id":"349426","messageId":"20180606003745.GI9266@aiede.svl.corp.google.com","threadId":"48648","inReplyTo":"c7e504a4664bb6008de970671af13eda7d85dc5b.1527894919.git.jonathantanmy@google.com","subject":"Re: [PATCH 6/6] fetch-pack: introduce negotiator API","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-06-06T00:37:45Z","receivedAt":"2018-06-06T00:37:51Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Tan wrote:\n\n> Introduce the new files fetch-negotiator.{h,c}, which contains an API\n> behind which the details of negotiation are abstracted. Currently, only\n> one algorithm is available: the existing one.\n>\n> This patch is written to be more easily reviewed: static functions are\n> moved verbatim from fetch-pack.c to negotiator/default.c, and it can be\n> seen that the lines replaced by negotiator->X() calls are present in the\n> X() functions respectively.\n\nnit: s/more//\n\n> Signed-off-by: Jonathan Tan \n> ---\n> Makefile | 2 +\n> fetch-negotiator.c | 7 ++\n> fetch-negotiator.h | 45 ++++++++++\n> fetch-pack.c | 207 ++++++++++---------------------------------\n> negotiator/default.c | 173 ++++++++++++++++++++++++++++++++++++\n> negotiator/default.h | 8 ++\n> object.h | 3 +-\n> 7 files changed, 282 insertions(+), 163 deletions(-)\n> create mode 100644 fetch-negotiator.c\n> create mode 100644 fetch-negotiator.h\n> create mode 100644 negotiator/default.c\n> create mode 100644 negotiator/default.h\n\nLooks like a job for --color-moved. :)\n\n[...]\n> --- /dev/null\n> +++ b/fetch-negotiator.c\n> @@ -0,0 +1,7 @@\n> +#include \"fetch-negotiator.h\"\n\nTo avoid various standards weirdness, the first #include in all files\nshould be git-compat-util.h, cache.h, or builtin.h. I tend to just\nuse git-compat-util.h.\n\n[...]\n> +++ b/fetch-negotiator.h\n> @@ -0,0 +1,45 @@\n> +#ifndef FETCH_NEGOTIATOR\n> +#define FETCH_NEGOTIATOR\n> +\n> +#include \"cache.h\"\n\nWhat does this need from cache.h?\n\n> +#include \"commit.h\"\n\noptional: could use a forward-declaration \"struct commit;\" since we\nonly use pointers instead of the complete type. Do we document when\nto prefer forward-declaration and when to #include complete type\ndefinitions somewhere?\n\n[...]\n> +struct fetch_negotiator {\n\nNeat. Can this file include an overview of the calling sequence / how\nI use this API? E.g.\n\n\t/*\n\t * Standard calling sequence:\n\t * 1. Obtain a struct fetch_negotiator * using [...]\n\t * 2. For each [etc]\n\t * 3. Free resources associated with the negotiator by calling\n\t * release(this). This frees the pointer; it cannot be\n\t * reused.\n\t */\n\nThat would be a good complement to the reference documentation in the\nstruct definition.\n\n[...]\n> +++ b/object.h\n> @@ -28,7 +28,8 @@ struct object_array {\n> /*\n> * object flag allocation:\n> * revision.h: 0---------10 26\n> - * fetch-pack.c: 0----5\n> + * fetch-pack.c: 01\n> + * negotiator/default.c: 2--5\n> * walker.c: 0-2\n\nNice!\n\n[...]\n> +++ b/fetch-pack.c\n[...]\n> @@ -462,7 +348,7 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n> \tretval = -1;\n> \tif (args->no_dependents)\n> \t\tgoto done;\n> -\twhile ((oid = get_rev(data))) {\n> +\twhile ((oid = negotiator->next(negotiator))) {\n[etc]\n\nI like it. :)\n\n[...]\n> @@ -988,7 +870,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n> \tstruct object_id oid;\n> \tconst char *agent_feature;\n> \tint agent_len;\n> -\tstruct data data = { { compare_commits_by_commit_date } };\n> +\tstruct fetch_negotiator negotiator;\n> +\tfetch_negotiator_init(&negotiator);\n\nSane.\n\n[...]\n> @@ -1061,19 +944,17 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n> \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n> \t\tdie(_(\"Server does not support --deepen\"));\n> \n> -\tif (marked)\n> -\t\tfor_each_ref(clear_marks, NULL);\n> -\tmarked = 1;\n> -\tif (everything_local(&data, args, &ref, sought, nr_sought)) {\n> +\tif (everything_local(&negotiator, args, &ref, sought, nr_sought)) {\n> \t\tpacket_flush(fd[1]);\n> \t\tgoto all_done;\n> \t}\n> -\tif (find_common(&data, args, fd, &oid, ref) < 0)\n> +\tif (find_common(&negotiator, args, fd, &oid, ref) < 0)\n> \t\tif (!args->keep_pack)\n> \t\t\t/* When cloning, it is not unusual to have\n> \t\t\t * no common commit.\n> \t\t\t */\n> \t\t\twarning(_(\"no common commits\"));\n> +\tnegotiator.release(&negotiator);\n\nShould this go after the all_done so that it doesn't get bypassed in\nthe everything_local case?\n\n[...]\n> @@ -1434,6 +1316,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\tcontinue;\n> \t\t}\n> \t}\n> +\tnegotiator.release(&negotiator);\n> \n> \toidset_clear(&common);\n\nnit: could put the negotiator.release(&negotiator) after the blank\nline, in the same paragraph as the other free calls like\noidset_clear(&common).\n\n[...]\n> +++ b/negotiator/default.c\n> @@ -0,0 +1,173 @@\n> +#include \"default.h\"\n\nFirst #include should be \"git-compat-util.h\".\n\n[...]\n> +/*\n> + This function marks a rev and its ancestors as common.\n> + In some cases, it is desirable to mark only the ancestors (for example\n> + when only the server does not yet know that they are common).\n> +*/\n\nNot about this change: comments should have ' *' at the start of each\nline (could do in a preparatory patch or a followup).\n\n[...]\n> --- /dev/null\n> +++ b/negotiator/default.h\n> @@ -0,0 +1,8 @@\n> +#ifndef NEGOTIATOR_DEFAULT_H\n> +#define NEGOTIATOR_DEFAULT_H\n> +\n> +#include \"fetch-negotiator.h\"\n> +\n> +void default_negotiator_init(struct fetch_negotiator *negotiator);\n\noptional: same question about whether to use a forward declaration as in\nfetch-negotiator.h.\n\nExcept where noted above,\nReviewed-by: Jonathan Nieder \n\nThanks for a nice cleanup.\n"},{"id":"349427","messageId":"20180605173818.4d56cf328d81b29a2878c1f6@google.com","threadId":"48648","inReplyTo":"20180605231634.GD9266@aiede.svl.corp.google.com","subject":"Re: [PATCH 2/6] fetch-pack: truly stop negotiation upon ACK ready","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T00:38:18Z","receivedAt":"2018-06-06T00:38:23Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 5 Jun 2018 16:16:34 -0700\nJonathan Nieder wrote:\n\n> Hi,\n> \n> Jonathan Tan wrote:\n> \n> > When \"ACK %s ready\" is received, find_common() clears rev_list in an\n> > attempt to stop further \"have\" lines from being sent [1]. This appears\n> > to work, despite the invocation to mark_common() in the \"while\" loop.\n> \n> Does \"appears to work\" mean \"works\" or \"doesn't work but does an okay\n> job of faking\"?\n\n\"Appears to work\" means I think that it works, but I don't think I can\nconclusively prove it.\n\n> > Though it is possible for mark_common() to invoke rev_list_push() (thus\n> > making rev_list non-empty once more), it is more likely that the commits\n> \n> nit: s/more likely/most likely/\n> or s/it is more likely that/usually/\n> \n> > in rev_list that have un-SEEN parents are also unparsed, meaning that\n> > mark_common() is not invoked on them.\n> >\n> > To avoid all this uncertainty, it is better to explicitly end the loop\n> > when \"ACK %s ready\" is received instead of clearing rev_list. Remove the\n> > clearing of rev_list and write \"if (got_ready) break;\" instead.\n> \n> I'm still a little curious about whether this can happen in practice\n> or whether it's just about readability (or whether you didn't figure\n> out which, which is perfectly fine), but either way it's a good\n> change.\n\nI tried to figure out which, but concluded that I can't.\n\nI think that in v2's commit message, I'll start with describing the\nreadability aspect.\n\n> > @@ -1281,7 +1281,6 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n> > \t\t}\n> > \n> > \t\tif (!strcmp(reader->line, \"ready\")) {\n> > -\t\t\tclear_prio_queue(&rev_list);\n> > \t\t\treceived_ready = 1;\n> > \t\t\tcontinue;\n> \n> I'm curious about the lifetime of &rev_list. Does the priority queue\n> get freed eventually?\n\nNo (which potentially causes a problem in the case that fetch-pack is\ninvoked twice), but I fix that in patch 4/6, so I didn't bother\naddressing it here. I'll add a note about the lifetime of this priority\nqueue in v2.\n"},{"id":"349428","messageId":"CAGf8dgLJy5yKx3yD3-ADdMTBW8_8N984wMN8FfN4g2_xjAON4A@mail.gmail.com","threadId":"48648","inReplyTo":"20180605233026.GF9266@aiede.svl.corp.google.com","subject":"Re: [PATCH 3/6] fetch-pack: in protocol v2, enqueue commons first","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T02:10:13Z","receivedAt":"2018-06-06T02:10:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, Jun 5, 2018 at 4:30 PM, Jonathan Nieder wrote:\n> I get lost in the above description. I suspect it's doing a good job\n> of describing the code, instead of answering the question I really\n> have about what is broken and what behavior we want instead.\n>\n> E.g. are there some commands that I can run to trigger the unnecessary\n> \"have\" lines? That would make it easier for me to understand the rest\n> and whether this is a good approach for suppressing them.\n>\n> It's possible I should be skipping to the test, but a summary in the\n> commit message would make life easier for lazy people like me. :)\n\nOK, I'll start the commit message with explaining a situation in which\nthese redundant \"have\" lines will appear instead. (The situation will\nbe the same as the one in the test.)\n\n> This is subtle. My instinct would be to assume that the purpose of\n> everything_local is just to determine whether we need to send a fetch\n> request, but it appears we also want to rely on a side effect from it.\n>\n> Could everything_local get a function comment to describe what side\n> effects we will be counting on from it?\n\nYou're right that there's a side effect in everything_local. In v2,\nI'll have a preparatory patch to separate it into a few functions so\nthat we can see what happens more clearly.\n\n> nit: this adds the new test as last in the script. Is there some\n> logical earlier place in the file it can go instead? That way, the\n> file stays organized and concurrent patches that modify the same test\n> script are less likely to conflict.\n\nGood point. I'll find a place.\n\n>> + rm -rf server client &&\n>> + git init server &&\n>> + test_commit -C server aref_both_1 &&\n>> + git -C server tag -d aref_both_1 &&\n>> + test_commit -C server aref_both_2 &&\n>\n> What does aref stand for?\n\n\"A ref\", \"a\" as in \"one\". I'll find a better name (probably just\n\"both_1\" and \"both_2\").\n\n>> +\n>> + # The ref name that only the server has must be a prefix of all the\n>> + # others, to ensure that the client has the same information regardless\n>> + # of whether protocol v0 (which does not have ref prefix filtering) or\n>> + # protocol v2 (which does) is used.\n>\n> must or else what? Maybe:\n>\n> # In this test, the ref name that only the server has is a prefix of\n> # all other refs. This ensures that the client has the same information\n> # regardless of [etc]\n\nThanks - I'll use your suggestion.\n\n>> + git clone server client &&\n>> + test_commit -C server aref &&\n>> + test_commit -C client aref_client &&\n>> +\n>> + # In both protocol v0 and v2, ensure that the parent of aref_both_2 is\n>> + # not sent as a \"have\" line.\n>\n> Why shouldn't it be sent as a \"have\" line? E.g. does another \"have\"\n> line make it redundant?\n\nThe server's ref advertisement makes it redundant. I'll explain this\nmore clearly in v2.\n\n>> +\n>> + rm -f trace &&\n>> + cp -r client clientv0 &&\n>> + GIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv0 \\\n>> + fetch origin aref &&\n>> + grep \"have $(git -C client rev-parse aref_client)\" trace &&\n>> + grep \"have $(git -C client rev-parse aref_both_2)\" trace &&\n>\n> nit: can make this more robust by doing\n>\n> aref_client=$(git -C client rev-parse aref_client) &&\n> aref_both_2=$(git -C client rev-parse aref_both_2) &&\n>\n> since this way if the git command fails, the test fails.\n\nWill do. Thanks for your comments.\n"},{"id":"349429","messageId":"CAGf8dgL2i2R50aRM9OoKGmVgS6XOSY+rD3F=RV8qd=Ozh=ONUA@mail.gmail.com","threadId":"48648","inReplyTo":"20180605233504.GG9266@aiede.svl.corp.google.com","subject":"Re: [PATCH 4/6] fetch-pack: make negotiation-related vars local","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T02:12:08Z","receivedAt":"2018-06-06T02:12:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, Jun 5, 2018 at 4:35 PM, Jonathan Nieder wrote:\n> Jonathan Tan wrote:\n>\n>> Reduce the number of global variables by making the priority queue and\n>> the count of non-common commits in it local, passing them as a struct to\n>> various functions where necessary.\n>\n> \\o/\n>\n>> This also helps in the case that fetch_pack() is invoked twice in the\n>> same process (when tag following is required when using a transport that\n>> does not support tag following), in that different priority queues will\n>> now be used in each invocation, instead of reusing the possibly\n>> non-empty one.\n>>\n>> The struct containing these variables is named \"data\" to ease review of\n>> a subsequent patch in this series - in that patch, this struct\n>> definition and several functions will be moved to a negotiation-specific\n>> file, and this allows the move to be verbatim.\n>\n> Hm. Is the idea that 'struct data' gets stored in the opaque 'data'\n> member of the fetch_negotiator?\n\nYes.\n\n> 'struct data' is a quite vague type name --- it's almost equivalent to\n> 'void' (which I suppose is the idea). How about something like\n> 'struct negotiation_data' or 'fetch_negotiator_data' in this patch?\n> That way this last paragraph of the commit message wouldn't be needed.\n\nI wanted to make it easier to review the subsequent patch, in that\nentire functions, including the signature, can be moved verbatim. If\nthe project prefers that I don't do that, let me know.\n\n> [...]\n>> --- a/fetch-pack.c\n>> +++ b/fetch-pack.c\n>> @@ -50,8 +50,12 @@ static int marked;\n>> */\n>> #define MAX_IN_VAIN 256\n>>\n>> -static struct prio_queue rev_list = { compare_commits_by_commit_date };\n>> -static int non_common_revs, multi_ack, use_sideband;\n>> +struct data {\n>> + struct prio_queue rev_list;\n>> + int non_common_revs;\n>> +};\n>\n> How does this struct get used? What does it represent? A comment\n> might help.\n\nI'll add a comment.\n"},{"id":"349430","messageId":"CAGf8dgLsP6MWSd9OTL7tq+92Rh8UA9Kitzn6nCiRj1bM+Eyk4Q@mail.gmail.com","threadId":"48648","inReplyTo":"20180606000146.GH9266@aiede.svl.corp.google.com","subject":"Re: [PATCH 5/6] fetch-pack: move common check and marking together","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T02:12:46Z","receivedAt":"2018-06-06T02:12:51Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, Jun 5, 2018 at 5:01 PM, Jonathan Nieder wrote:\n> I like it. I think it should be possible to describe the benefit of\n> this patch without reference to the specifics of the subsequent one.\n> Maybe something like:\n>\n> When receiving 'ACK continue' for a common commit,\n> check if the commit was already known to be common and mark it\n> as such if not up front. This should make future refactoring\n> of how the information about common commits is stored more\n> straightforward.\n>\n> No visible change intended.\n\nThanks, I'll use your suggestion.\n"},{"id":"349431","messageId":"CAGf8dgKWPy6ACsNPas=z1dT6qirAEHkkk8wCo91cFaxPVkgrhQ@mail.gmail.com","threadId":"48648","inReplyTo":"20180606003745.GI9266@aiede.svl.corp.google.com","subject":"Re: [PATCH 6/6] fetch-pack: introduce negotiator API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T02:17:15Z","receivedAt":"2018-06-06T02:17:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, Jun 5, 2018 at 5:37 PM, Jonathan Nieder wrote:\n>> This patch is written to be more easily reviewed: static functions are\n>> moved verbatim from fetch-pack.c to negotiator/default.c, and it can be\n>> seen that the lines replaced by negotiator->X() calls are present in the\n>> X() functions respectively.\n>\n> nit: s/more//\n\nOK.\n\n>> +#include \"fetch-negotiator.h\"\n>\n> To avoid various standards weirdness, the first #include in all files\n> should be git-compat-util.h, cache.h, or builtin.h. I tend to just\n> use git-compat-util.h.\n\nOK.\n\n>> +#include \"cache.h\"\n>\n> What does this need from cache.h?\n\nIf I remember correctly, I needed something, but I might not. I'll\nremove it if so.\n\n>> +#include \"commit.h\"\n>\n> optional: could use a forward-declaration \"struct commit;\" since we\n> only use pointers instead of the complete type. Do we document when\n> to prefer forward-declaration and when to #include complete type\n> definitions somewhere?\n\nI'll use the forward declaration then.\n\n> [...]\n>> +struct fetch_negotiator {\n>\n> Neat. Can this file include an overview of the calling sequence / how\n> I use this API? E.g.\n>\n> /*\n> * Standard calling sequence:\n> * 1. Obtain a struct fetch_negotiator * using [...]\n> * 2. For each [etc]\n> * 3. Free resources associated with the negotiator by calling\n> * release(this). This frees the pointer; it cannot be\n> * reused.\n> */\n>\n> That would be a good complement to the reference documentation in the\n> struct definition.\n\nOK.\n\n>> @@ -1061,19 +944,17 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n>> if (!server_supports(\"deepen-relative\") && args->deepen_relative)\n>> die(_(\"Server does not support --deepen\"));\n>>\n>> - if (marked)\n>> - for_each_ref(clear_marks, NULL);\n>> - marked = 1;\n>> - if (everything_local(&data, args, &ref, sought, nr_sought)) {\n>> + if (everything_local(&negotiator, args, &ref, sought, nr_sought)) {\n>> packet_flush(fd[1]);\n>> goto all_done;\n>> }\n>> - if (find_common(&data, args, fd, &oid, ref) < 0)\n>> + if (find_common(&negotiator, args, fd, &oid, ref) < 0)\n>> if (!args->keep_pack)\n>> /* When cloning, it is not unusual to have\n>> * no common commit.\n>> */\n>> warning(_(\"no common commits\"));\n>> + negotiator.release(&negotiator);\n>\n> Should this go after the all_done so that it doesn't get bypassed in\n> the everything_local case?\n\nGood point - will do.\n\n>> @@ -1434,6 +1316,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n>> continue;\n>> }\n>> }\n>> + negotiator.release(&negotiator);\n>>\n>> oidset_clear(&common);\n>\n> nit: could put the negotiator.release(&negotiator) after the blank\n> line, in the same paragraph as the other free calls like\n> oidset_clear(&common).\n\nOK.\n\n>> +++ b/negotiator/default.c\n>> @@ -0,0 +1,173 @@\n>> +#include \"default.h\"\n>\n> First #include should be \"git-compat-util.h\".\n\nOK.\n\n> [...]\n>> +/*\n>> + This function marks a rev and its ancestors as common.\n>> + In some cases, it is desirable to mark only the ancestors (for example\n>> + when only the server does not yet know that they are common).\n>> +*/\n>\n> Not about this change: comments should have ' *' at the start of each\n> line (could do in a preparatory patch or a followup).\n\nI'll add a followup.\n\n>> --- /dev/null\n>> +++ b/negotiator/default.h\n>> @@ -0,0 +1,8 @@\n>> +#ifndef NEGOTIATOR_DEFAULT_H\n>> +#define NEGOTIATOR_DEFAULT_H\n>> +\n>> +#include \"fetch-negotiator.h\"\n>> +\n>> +void default_negotiator_init(struct fetch_negotiator *negotiator);\n>\n> optional: same question about whether to use a forward declaration as in\n> fetch-negotiator.h.\n\nI'll use a forward declaration.\n"},{"id":"349556","messageId":"cover.1528317619.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1527894919.git.jonathantanmy@google.com","subject":"[PATCH v2 0/8] Refactor fetch negotiation into its own API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T20:47:06Z","receivedAt":"2018-06-06T20:47:27Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks, Jonathan Nieder, for your comments.\n\nThis patch set now includes a patch at the beginning to split up\neverything_local(), making it clear which functions have side effects\nand which don't, and a patch at the end to reformat some comments.\n\nWhile I was implementing the suggestions, there were some that I were\nunsure about. Here they are:\n\n> > nit: this adds the new test as last in the script. Is there some\n> > logical earlier place in the file it can go instead? That way, the\n> > file stays organized and concurrent patches that modify the same test\n> > script are less likely to conflict.\n> \n> Good point. I'll find a place.\n\nI couldn't find a logical place (I didn't see any existing tests that\ntest negotiation). I did move it up a bit earlier in the file, before\nfiltering by blob size, because that seemed more distinct from the rest\nof the tests.\n\n> >> -static struct prio_queue rev_list = { compare_commits_by_commit_date };\n> >> -static int non_common_revs, multi_ack, use_sideband;\n> >> +struct data {\n> >> + struct prio_queue rev_list;\n> >> + int non_common_revs;\n> >> +};\n> >\n> > How does this struct get used? What does it represent? A comment\n> > might help.\n> \n> I'll add a comment.\n\nI thought of adding a comment, but felt that I would end up removing it\nanyway upon its move to negotiator/default.c, so I didn't end up adding\nit.\n\n> >> +/*\n> >> + This function marks a rev and its ancestors as common.\n> >> + In some cases, it is desirable to mark only the ancestors (for example\n> >> + when only the server does not yet know that they are common).\n> >> +*/\n> >\n> > Not about this change: comments should have ' *' at the start of each\n> > line (could do in a preparatory patch or a followup).\n> \n> I'll add a followup.\n\nI'm now not sure of the value of making a change just to update\nformatting, but I added the followup commit anyway - it can be easily\ndropped if we decide to do so.\n\nJonathan Tan (8):\n fetch-pack: split up everything_local()\n fetch-pack: clear marks before re-marking\n fetch-pack: directly end negotiation if ACK ready\n fetch-pack: use ref adv. to prune \"have\" sent\n fetch-pack: make negotiation-related vars local\n fetch-pack: move common check and marking together\n fetch-pack: introduce negotiator API\n negotiator/default: use better style in comments\n\n Makefile | 2 +\n fetch-negotiator.c | 8 ++\n fetch-negotiator.h | 57 ++++++++++\n fetch-pack.c | 254 ++++++++++++++----------------------------\n negotiator/default.c | 174 +++++++++++++++++++++++++++++\n negotiator/default.h | 8 ++\n object.h | 3 +-\n t/t5500-fetch-pack.sh | 39 +++++++\n 8 files changed, 376 insertions(+), 169 deletions(-)\n create mode 100644 fetch-negotiator.c\n create mode 100644 fetch-negotiator.h\n create mode 100644 negotiator/default.c\n create mode 100644 negotiator/default.h\n\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349557","messageId":"5687c0b22bf3b2bdbbe29b09788cc305a89710fc.1528317619.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1528317619.git.jonathantanmy@google.com","subject":"[PATCH v2 1/8] fetch-pack: split up everything_local()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T20:47:07Z","receivedAt":"2018-06-06T20:47:29Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function everything_local(), despite its name, also (1) marks\ncommits as COMPLETE and COMMON_REF and (2) invokes filter_refs() as\nimportant side effects. Extract (1) into its own function\n(mark_complete_and_common_ref()) and remove\n(2).\n\nThe restoring of save_commit_buffer, which was introduced in a1c6d7c1a7\n(\"fetch-pack: restore save_commit_buffer after use\", 2017-12-08), is a\nconcern of the parse_object() call in mark_complete_and_common_ref(), so\nit has been moved from the end of everything_local() to the end of\nmark_complete_and_common_ref().\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 39 ++++++++++++++++++++++++++++++---------\n 1 file changed, 30 insertions(+), 9 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex a320ce987..5c87bb8bb 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -734,12 +734,20 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n \treturn 0;\n }\n \n-static int everything_local(struct fetch_pack_args *args,\n-\t\t\t struct ref **refs,\n-\t\t\t struct ref **sought, int nr_sought)\n+/*\n+ * Mark recent commits available locally and reachable from a local ref as\n+ * COMPLETE. If args->no_dependents is false, also mark COMPLETE remote refs as\n+ * COMMON_REF (otherwise, we are not planning to participate in negotiation, and\n+ * thus do not need COMMON_REF marks).\n+ *\n+ * The cutoff time for recency is determined by this heuristic: it is the\n+ * earliest commit time of the objects in refs that are commits and that we know\n+ * the commit time of.\n+ */\n+static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n+\t\t\t\t\t struct ref **refs)\n {\n \tstruct ref *ref;\n-\tint retval;\n \tint old_save_commit_buffer = save_commit_buffer;\n \ttimestamp_t cutoff = 0;\n \tstruct oidset loose_oid_set = OIDSET_INIT;\n@@ -812,7 +820,18 @@ static int everything_local(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n-\tfilter_refs(args, refs, sought, nr_sought);\n+\tsave_commit_buffer = old_save_commit_buffer;\n+}\n+\n+/*\n+ * Returns 1 if every object pointed to by the given remote refs is available\n+ * locally and reachable from a local ref, and 0 otherwise.\n+ */\n+static int everything_local(struct fetch_pack_args *args,\n+\t\t\t struct ref **refs)\n+{\n+\tstruct ref *ref;\n+\tint retval;\n \n \tfor (retval = 1, ref = *refs; ref ; ref = ref->next) {\n \t\tconst struct object_id *remote = &ref->old_oid;\n@@ -829,8 +848,6 @@ static int everything_local(struct fetch_pack_args *args,\n \t\t\t ref->name);\n \t}\n \n-\tsave_commit_buffer = old_save_commit_buffer;\n-\n \treturn retval;\n }\n \n@@ -1053,7 +1070,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n \n-\tif (everything_local(args, &ref, sought, nr_sought)) {\n+\tmark_complete_and_common_ref(args, &ref);\n+\tfilter_refs(args, &ref, sought, nr_sought);\n+\tif (everything_local(args, &ref)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n@@ -1377,7 +1396,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n \n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n-\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n+\t\t\tmark_complete_and_common_ref(args, &ref);\n+\t\t\tfilter_refs(args, &ref, sought, nr_sought);\n+\t\t\tif (everything_local(args, &ref))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349558","messageId":"dca6619c551b8259aef0476146184996657a80e9.1528317619.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1528317619.git.jonathantanmy@google.com","subject":"[PATCH v2 2/8] fetch-pack: clear marks before re-marking","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T20:47:08Z","receivedAt":"2018-06-06T20:47:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"If tag following is required when using a transport that does not\nsupport tag following, fetch_pack() will be invoked twice in the same\nprocess, necessitating a clearing of the object flags used by\nfetch_pack() sometime during the second invocation. This is currently\ndone in find_common(), which means that the invocation of\nmark_complete_and_common_ref() in do_fetch_pack() is useless.\n\n(This cannot be reproduced with Git alone, because all transports that\ncome with Git support tag following.)\n\nTherefore, move this clearing from find_common() to its\nparent function do_fetch_pack(), right before it calls\nmark_complete_and_common_ref().\n\nThis has been occurring since the commit that introduced the clearing of\nmarks, 420e9af498 (\"Fix tag following\", 2008-03-19).\n\nThe corresponding code for protocol v2 in do_fetch_pack_v2() does not\nhave this problem, as the clearing of flags is done before any marking\n(whether by rev_list_insert_ref_oid() or\nmark_complete_and_common_ref()).\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 5c87bb8bb..2812499a5 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -336,9 +336,6 @@ static int find_common(struct fetch_pack_args *args,\n \n \tif (args->stateless_rpc && multi_ack == 1)\n \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n-\tif (marked)\n-\t\tfor_each_ref(clear_marks, NULL);\n-\tmarked = 1;\n \n \tfor_each_ref(rev_list_insert_ref_oid, NULL);\n \tfor_each_cached_alternate(insert_one_alternate_object);\n@@ -1070,6 +1067,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n \n+\tif (marked)\n+\t\tfor_each_ref(clear_marks, NULL);\n+\tmarked = 1;\n \tmark_complete_and_common_ref(args, &ref);\n \tfilter_refs(args, &ref, sought, nr_sought);\n \tif (everything_local(args, &ref)) {\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349559","messageId":"f12342fb2760eb0449c86c66bf44d39f5871be57.1528317619.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1528317619.git.jonathantanmy@google.com","subject":"[PATCH v2 4/8] fetch-pack: use ref adv. to prune \"have\" sent","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T20:47:10Z","receivedAt":"2018-06-06T20:47:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In negotiation using protocol v2, fetch-pack sometimes does not make\nfull use of the information obtained in the ref advertisement:\nspecifically, that if the server advertises a commit that the client\nalso has, the client never needs to inform the server that it has the\ncommit's parents, since it can just tell the server that it has the\nadvertised commit and it knows that the server can and will infer the\nrest.\n\nThis is because, in do_fetch_pack_v2(), rev_list_insert_ref_oid() is\ninvoked before everything_local(). This means that if we have a commit\nthat is both our ref and their ref, it would be enqueued by\nrev_list_insert_ref_oid() as SEEN, and since it is thus already SEEN,\neverything_local() would not enqueue it.\n\nIf everything_local() were invoked first, as it is in do_fetch_pack()\nfor protocol v0, then everything_local() would enqueue it with\nCOMMON_REF | SEEN. The addition of COMMON_REF ensures that its parents\nare not sent as \"have\" lines.\n\nChange the order in do_fetch_pack_v2() to be consistent with\ndo_fetch_pack(), and to avoid sending unnecessary \"have\" lines.\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 6 +++---\n t/t5500-fetch-pack.sh | 39 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 42 insertions(+), 3 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 09f5c83c4..114207b8e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1391,9 +1391,6 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t\tfor_each_ref(clear_marks, NULL);\n \t\t\tmarked = 1;\n \n-\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n-\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tmark_complete_and_common_ref(args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n@@ -1401,6 +1398,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n+\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n+\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n \t\t\tbreak;\n \t\tcase FETCH_SEND_REQUEST:\n \t\t\tif (send_fetch_request(fd[1], args, ref, &common,\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex d4f435155..026ba9c9e 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -755,6 +755,45 @@ test_expect_success 'fetching deepen' '\n \t)\n '\n \n+test_expect_success 'use ref advertisement to prune \"have\" lines sent' '\n+\trm -rf server client &&\n+\tgit init server &&\n+\ttest_commit -C server both_have_1 &&\n+\tgit -C server tag -d both_have_1 &&\n+\ttest_commit -C server both_have_2 &&\n+\n+\t# In this test, the ref name that only the server has is a prefix of all\n+\t# other refs. This is because in protocol v2, the client sends\n+\t# \"ref-prefix\" to limit the ref advertisement. Naming the ref \"bo\" means\n+\t# that \"ref-prefix refs/tags/bo*\" is sent, resulting in the client also\n+\t# knowing about refs/tags/both_have_2, just as it would when it uses\n+\t# protocol v0.\n+\tgit clone server client &&\n+\ttest_commit -C server bo &&\n+\ttest_commit -C client client_has &&\n+\n+\t# In both protocol v0 and v2, ensure that the parent of both_have_2 is\n+\t# not sent as a \"have\" line. The client should know that the server has\n+\t# both_have_2, so it only needs to inform the server that it has\n+\t# both_have_2, and the server can infer the rest.\n+\n+\trm -f trace &&\n+\tcp -r client clientv0 &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv0 \\\n+\t\tfetch origin bo &&\n+\tgrep \"have $(git -C client rev-parse client_has)\" trace &&\n+\tgrep \"have $(git -C client rev-parse both_have_2)\" trace &&\n+\t! grep \"have $(git -C client rev-parse both_have_2^)\" trace &&\n+\n+\trm -f trace &&\n+\tcp -r client clientv2 &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv2 -c protocol.version=2 \\\n+\t\tfetch origin bo &&\n+\tgrep \"have $(git -C client rev-parse client_has)\" trace &&\n+\tgrep \"have $(git -C client rev-parse both_have_2)\" trace &&\n+\t! grep \"have $(git -C client rev-parse both_have_2^)\" trace\n+'\n+\n test_expect_success 'filtering by size' '\n \trm -rf server client &&\n \ttest_create_repo server &&\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349560","messageId":"955d9f62d2c8400791501d71c72fea4ef2dc1cff.1528317619.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1528317619.git.jonathantanmy@google.com","subject":"[PATCH v2 5/8] fetch-pack: make negotiation-related vars local","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T20:47:11Z","receivedAt":"2018-06-06T20:47:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Reduce the number of global variables by making the priority queue and\nthe count of non-common commits in it local, passing them as a struct to\nvarious functions where necessary.\n\nThis also helps in the case that fetch_pack() is invoked twice in the\nsame process (when tag following is required when using a transport that\ndoes not support tag following), in that different priority queues will\nnow be used in each invocation, instead of reusing the possibly\nnon-empty one.\n\nThe struct containing these variables is named \"data\" to ease review of\na subsequent patch in this series - in that patch, this struct\ndefinition and several functions will be moved to a negotiation-specific\nfile, and this allows the move to be verbatim.\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 104 +++++++++++++++++++++++++++++----------------------\n 1 file changed, 59 insertions(+), 45 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 114207b8e..c31644bb9 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -50,8 +50,12 @@ static int marked;\n */\n #define MAX_IN_VAIN 256\n \n-static struct prio_queue rev_list = { compare_commits_by_commit_date };\n-static int non_common_revs, multi_ack, use_sideband;\n+struct data {\n+\tstruct prio_queue rev_list;\n+\tint non_common_revs;\n+};\n+\n+static int multi_ack, use_sideband;\n /* Allow specifying sha1 if it is a ref tip. */\n #define ALLOW_TIP_SHA1\t01\n /* Allow request of a sha1 if it is reachable from a ref (possibly hidden ref). */\n@@ -93,7 +97,8 @@ static void cache_one_alternate(const char *refname,\n \tcache->items[cache->nr++] = obj;\n }\n \n-static void for_each_cached_alternate(void (*cb)(struct object *))\n+static void for_each_cached_alternate(struct data *data,\n+\t\t\t\t void (*cb)(struct data *, struct object *))\n {\n \tstatic int initialized;\n \tstatic struct alternate_object_cache cache;\n@@ -105,10 +110,10 @@ static void for_each_cached_alternate(void (*cb)(struct object *))\n \t}\n \n \tfor (i = 0; i < cache.nr; i++)\n-\t\tcb(cache.items[i]);\n+\t\tcb(data, cache.items[i]);\n }\n \n-static void rev_list_push(struct commit *commit, int mark)\n+static void rev_list_push(struct data *data, struct commit *commit, int mark)\n {\n \tif (!(commit->object.flags & mark)) {\n \t\tcommit->object.flags |= mark;\n@@ -116,19 +121,20 @@ static void rev_list_push(struct commit *commit, int mark)\n \t\tif (parse_commit(commit))\n \t\t\treturn;\n \n-\t\tprio_queue_put(&rev_list, commit);\n+\t\tprio_queue_put(&data->rev_list, commit);\n \n \t\tif (!(commit->object.flags & COMMON))\n-\t\t\tnon_common_revs++;\n+\t\t\tdata->non_common_revs++;\n \t}\n }\n \n-static int rev_list_insert_ref(const char *refname, const struct object_id *oid)\n+static int rev_list_insert_ref(struct data *data, const char *refname,\n+\t\t\t const struct object_id *oid)\n {\n \tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n \n \tif (o && o->type == OBJ_COMMIT)\n-\t\trev_list_push((struct commit *)o, SEEN);\n+\t\trev_list_push(data, (struct commit *)o, SEEN);\n \n \treturn 0;\n }\n@@ -136,7 +142,7 @@ static int rev_list_insert_ref(const char *refname, const struct object_id *oid)\n static int rev_list_insert_ref_oid(const char *refname, const struct object_id *oid,\n \t\t\t\t int flag, void *cb_data)\n {\n-\treturn rev_list_insert_ref(refname, oid);\n+\treturn rev_list_insert_ref(cb_data, refname, oid);\n }\n \n static int clear_marks(const char *refname, const struct object_id *oid,\n@@ -156,7 +162,7 @@ static int clear_marks(const char *refname, const struct object_id *oid,\n when only the server does not yet know that they are common).\n */\n \n-static void mark_common(struct commit *commit,\n+static void mark_common(struct data *data, struct commit *commit,\n \t\tint ancestors_only, int dont_parse)\n {\n \tif (commit != NULL && !(commit->object.flags & COMMON)) {\n@@ -166,12 +172,12 @@ static void mark_common(struct commit *commit,\n \t\t\to->flags |= COMMON;\n \n \t\tif (!(o->flags & SEEN))\n-\t\t\trev_list_push(commit, SEEN);\n+\t\t\trev_list_push(data, commit, SEEN);\n \t\telse {\n \t\t\tstruct commit_list *parents;\n \n \t\t\tif (!ancestors_only && !(o->flags & POPPED))\n-\t\t\t\tnon_common_revs--;\n+\t\t\t\tdata->non_common_revs--;\n \t\t\tif (!o->parsed && !dont_parse)\n \t\t\t\tif (parse_commit(commit))\n \t\t\t\t\treturn;\n@@ -179,7 +185,7 @@ static void mark_common(struct commit *commit,\n \t\t\tfor (parents = commit->parents;\n \t\t\t\t\tparents;\n \t\t\t\t\tparents = parents->next)\n-\t\t\t\tmark_common(parents->item, 0, dont_parse);\n+\t\t\t\tmark_common(data, parents->item, 0, dont_parse);\n \t\t}\n \t}\n }\n@@ -188,7 +194,7 @@ static void mark_common(struct commit *commit,\n Get the next rev to send, ignoring the common.\n */\n \n-static const struct object_id *get_rev(void)\n+static const struct object_id *get_rev(struct data *data)\n {\n \tstruct commit *commit = NULL;\n \n@@ -196,16 +202,16 @@ static const struct object_id *get_rev(void)\n \t\tunsigned int mark;\n \t\tstruct commit_list *parents;\n \n-\t\tif (rev_list.nr == 0 || non_common_revs == 0)\n+\t\tif (data->rev_list.nr == 0 || data->non_common_revs == 0)\n \t\t\treturn NULL;\n \n-\t\tcommit = prio_queue_get(&rev_list);\n+\t\tcommit = prio_queue_get(&data->rev_list);\n \t\tparse_commit(commit);\n \t\tparents = commit->parents;\n \n \t\tcommit->object.flags |= POPPED;\n \t\tif (!(commit->object.flags & COMMON))\n-\t\t\tnon_common_revs--;\n+\t\t\tdata->non_common_revs--;\n \n \t\tif (commit->object.flags & COMMON) {\n \t\t\t/* do not send \"have\", and ignore ancestors */\n@@ -220,9 +226,9 @@ static const struct object_id *get_rev(void)\n \n \t\twhile (parents) {\n \t\t\tif (!(parents->item->object.flags & SEEN))\n-\t\t\t\trev_list_push(parents->item, mark);\n+\t\t\t\trev_list_push(data, parents->item, mark);\n \t\t\tif (mark & COMMON)\n-\t\t\t\tmark_common(parents->item, 1, 0);\n+\t\t\t\tmark_common(data, parents->item, 1, 0);\n \t\t\tparents = parents->next;\n \t\t}\n \t}\n@@ -296,9 +302,9 @@ static void send_request(struct fetch_pack_args *args,\n \t\twrite_or_die(fd, buf->buf, buf->len);\n }\n \n-static void insert_one_alternate_object(struct object *obj)\n+static void insert_one_alternate_object(struct data *data, struct object *obj)\n {\n-\trev_list_insert_ref(NULL, &obj->oid);\n+\trev_list_insert_ref(data, NULL, &obj->oid);\n }\n \n #define INITIAL_FLUSH 16\n@@ -321,7 +327,7 @@ static int next_flush(int stateless_rpc, int count)\n \treturn count;\n }\n \n-static int find_common(struct fetch_pack_args *args,\n+static int find_common(struct data *data, struct fetch_pack_args *args,\n \t\t int fd[2], struct object_id *result_oid,\n \t\t struct ref *refs)\n {\n@@ -337,8 +343,8 @@ static int find_common(struct fetch_pack_args *args,\n \tif (args->stateless_rpc && multi_ack == 1)\n \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n \n-\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\tfor_each_cached_alternate(insert_one_alternate_object);\n+\tfor_each_ref(rev_list_insert_ref_oid, data);\n+\tfor_each_cached_alternate(data, insert_one_alternate_object);\n \n \tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\n@@ -456,7 +462,7 @@ static int find_common(struct fetch_pack_args *args,\n \tretval = -1;\n \tif (args->no_dependents)\n \t\tgoto done;\n-\twhile ((oid = get_rev())) {\n+\twhile ((oid = get_rev(data))) {\n \t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tprint_verbose(args, \"have %s\", oid_to_hex(oid));\n \t\tin_vain++;\n@@ -514,7 +520,7 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\t\t\t} else if (!args->stateless_rpc\n \t\t\t\t\t\t || ack != ACK_common)\n \t\t\t\t\t\tin_vain = 0;\n-\t\t\t\t\tmark_common(commit, 0, 1);\n+\t\t\t\t\tmark_common(data, commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgot_continue = 1;\n \t\t\t\t\tif (ack == ACK_ready)\n@@ -704,7 +710,7 @@ static void filter_refs(struct fetch_pack_args *args,\n \t*refs = newlist;\n }\n \n-static void mark_alternate_complete(struct object *obj)\n+static void mark_alternate_complete(struct data *unused, struct object *obj)\n {\n \tmark_complete(&obj->oid);\n }\n@@ -741,7 +747,8 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n * earliest commit time of the objects in refs that are commits and that we know\n * the commit time of.\n */\n-static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n+static void mark_complete_and_common_ref(struct data *data,\n+\t\t\t\t\t struct fetch_pack_args *args,\n \t\t\t\t\t struct ref **refs)\n {\n \tstruct ref *ref;\n@@ -792,7 +799,7 @@ static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n \tif (!args->no_dependents) {\n \t\tif (!args->deepen) {\n \t\t\tfor_each_ref(mark_complete_oid, NULL);\n-\t\t\tfor_each_cached_alternate(mark_alternate_complete);\n+\t\t\tfor_each_cached_alternate(NULL, mark_alternate_complete);\n \t\t\tcommit_list_sort_by_date(&complete);\n \t\t\tif (cutoff)\n \t\t\t\tmark_recent_complete_commits(args, cutoff);\n@@ -810,9 +817,10 @@ static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n \t\t\t\tcontinue;\n \n \t\t\tif (!(o->flags & SEEN)) {\n-\t\t\t\trev_list_push((struct commit *)o, COMMON_REF | SEEN);\n+\t\t\t\trev_list_push(data, (struct commit *)o,\n+\t\t\t\t\t COMMON_REF | SEEN);\n \n-\t\t\t\tmark_common((struct commit *)o, 1, 1);\n+\t\t\t\tmark_common(data, (struct commit *)o, 1, 1);\n \t\t\t}\n \t\t}\n \t}\n@@ -995,6 +1003,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tstruct object_id oid;\n \tconst char *agent_feature;\n \tint agent_len;\n+\tstruct data data = { { compare_commits_by_commit_date } };\n \n \tsort_ref_list(&ref, ref_compare_name);\n \tQSORT(sought, nr_sought, cmp_ref_by_name);\n@@ -1070,13 +1079,13 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (marked)\n \t\tfor_each_ref(clear_marks, NULL);\n \tmarked = 1;\n-\tmark_complete_and_common_ref(args, &ref);\n+\tmark_complete_and_common_ref(&data, args, &ref);\n \tfilter_refs(args, &ref, sought, nr_sought);\n \tif (everything_local(args, &ref)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n-\tif (find_common(args, fd, &oid, ref) < 0)\n+\tif (find_common(&data, args, fd, &oid, ref) < 0)\n \t\tif (!args->keep_pack)\n \t\t\t/* When cloning, it is not unusual to have\n \t\t\t * no common commit.\n@@ -1157,13 +1166,14 @@ static void add_common(struct strbuf *req_buf, struct oidset *common)\n \t}\n }\n \n-static int add_haves(struct strbuf *req_buf, int *haves_to_send, int *in_vain)\n+static int add_haves(struct data *data, struct strbuf *req_buf,\n+\t\t int *haves_to_send, int *in_vain)\n {\n \tint ret = 0;\n \tint haves_added = 0;\n \tconst struct object_id *oid;\n \n-\twhile ((oid = get_rev())) {\n+\twhile ((oid = get_rev(data))) {\n \t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tif (++haves_added >= *haves_to_send)\n \t\t\tbreak;\n@@ -1182,7 +1192,8 @@ static int add_haves(struct strbuf *req_buf, int *haves_to_send, int *in_vain)\n \treturn ret;\n }\n \n-static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n+static int send_fetch_request(struct data *data, int fd_out,\n+\t\t\t const struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain)\n {\n@@ -1238,7 +1249,7 @@ static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n \t\tadd_common(&req_buf, common);\n \n \t\t/* Add initial haves */\n-\t\tret = add_haves(&req_buf, haves_to_send, in_vain);\n+\t\tret = add_haves(data, &req_buf, haves_to_send, in_vain);\n \t}\n \n \t/* Send request */\n@@ -1275,7 +1286,8 @@ static int process_section_header(struct packet_reader *reader,\n \treturn ret;\n }\n \n-static int process_acks(struct packet_reader *reader, struct oidset *common)\n+static int process_acks(struct data *data, struct packet_reader *reader,\n+\t\t\tstruct oidset *common)\n {\n \t/* received */\n \tint received_ready = 0;\n@@ -1294,7 +1306,7 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n \t\t\t\tstruct commit *commit;\n \t\t\t\toidset_insert(common, &oid);\n \t\t\t\tcommit = lookup_commit(&oid);\n-\t\t\t\tmark_common(commit, 0, 1);\n+\t\t\t\tmark_common(data, commit, 0, 1);\n \t\t\t}\n \t\t\tcontinue;\n \t\t}\n@@ -1372,6 +1384,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tstruct packet_reader reader;\n \tint in_vain = 0;\n \tint haves_to_send = INITIAL_FLUSH;\n+\tstruct data data = { { compare_commits_by_commit_date } };\n \tpacket_reader_init(&reader, fd[0], NULL, 0,\n \t\t\t PACKET_READ_CHOMP_NEWLINE);\n \n@@ -1392,18 +1405,19 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tmarked = 1;\n \n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n-\t\t\tmark_complete_and_common_ref(args, &ref);\n+\t\t\tmark_complete_and_common_ref(&data, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n \t\t\tif (everything_local(args, &ref))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n \n-\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, &data);\n+\t\t\tfor_each_cached_alternate(&data,\n+\t\t\t\t\t\t insert_one_alternate_object);\n \t\t\tbreak;\n \t\tcase FETCH_SEND_REQUEST:\n-\t\t\tif (send_fetch_request(fd[1], args, ref, &common,\n+\t\t\tif (send_fetch_request(&data, fd[1], args, ref, &common,\n \t\t\t\t\t &haves_to_send, &in_vain))\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\telse\n@@ -1411,7 +1425,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tbreak;\n \t\tcase FETCH_PROCESS_ACKS:\n \t\t\t/* Process ACKs/NAKs */\n-\t\t\tswitch (process_acks(&reader, &common)) {\n+\t\t\tswitch (process_acks(&data, &reader, &common)) {\n \t\t\tcase 2:\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\t\tbreak;\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349561","messageId":"40f3e50ad41ecddf33b3d1b68f507f9e76ab0517.1528317619.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1528317619.git.jonathantanmy@google.com","subject":"[PATCH v2 6/8] fetch-pack: move common check and marking together","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T20:47:12Z","receivedAt":"2018-06-06T20:47:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"When receiving 'ACK continue' for a common commit, check if\nthe commit was already known to be common and mark it as such if not up\nfront. This should make future refactoring of how the information about\ncommon commits is stored more straightforward.\n\nNo visible change intended.\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex c31644bb9..4a4ec4da3 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -499,11 +499,14 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \t\t\t\tcase ACK_continue: {\n \t\t\t\t\tstruct commit *commit =\n \t\t\t\t\t\tlookup_commit(result_oid);\n+\t\t\t\t\tint was_common;\n \t\t\t\t\tif (!commit)\n \t\t\t\t\t\tdie(_(\"invalid commit %s\"), oid_to_hex(result_oid));\n+\t\t\t\t\twas_common = commit->object.flags & COMMON;\n+\t\t\t\t\tmark_common(data, commit, 0, 1);\n \t\t\t\t\tif (args->stateless_rpc\n \t\t\t\t\t && ack == ACK_common\n-\t\t\t\t\t && !(commit->object.flags & COMMON)) {\n+\t\t\t\t\t && !was_common) {\n \t\t\t\t\t\t/* We need to replay the have for this object\n \t\t\t\t\t\t * on the next RPC request so the peer knows\n \t\t\t\t\t\t * it is in common with us.\n@@ -520,7 +523,6 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \t\t\t\t\t} else if (!args->stateless_rpc\n \t\t\t\t\t\t || ack != ACK_common)\n \t\t\t\t\t\tin_vain = 0;\n-\t\t\t\t\tmark_common(data, commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgot_continue = 1;\n \t\t\t\t\tif (ack == ACK_ready)\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349562","messageId":"aa0df5f449c2f5c1f2e6d6edf933aad5f58b2a5e.1528317619.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1528317619.git.jonathantanmy@google.com","subject":"[PATCH v2 8/8] negotiator/default: use better style in comments","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T20:47:14Z","receivedAt":"2018-06-06T20:47:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n negotiator/default.c | 14 ++++++--------\n 1 file changed, 6 insertions(+), 8 deletions(-)\n\ndiff --git a/negotiator/default.c b/negotiator/default.c\nindex b8f45cf78..a9e52c943 100644\n--- a/negotiator/default.c\n+++ b/negotiator/default.c\n@@ -46,11 +46,10 @@ static int clear_marks(const char *refname, const struct object_id *oid,\n }\n \n /*\n- This function marks a rev and its ancestors as common.\n- In some cases, it is desirable to mark only the ancestors (for example\n- when only the server does not yet know that they are common).\n-*/\n-\n+ * This function marks a rev and its ancestors as common.\n+ * In some cases, it is desirable to mark only the ancestors (for example\n+ * when only the server does not yet know that they are common).\n+ */\n static void mark_common(struct data *data, struct commit *commit,\n \t\tint ancestors_only, int dont_parse)\n {\n@@ -80,9 +79,8 @@ static void mark_common(struct data *data, struct commit *commit,\n }\n \n /*\n- Get the next rev to send, ignoring the common.\n-*/\n-\n+ * Get the next rev to send, ignoring the common.\n+ */\n static const struct object_id *get_rev(struct data *data)\n {\n \tstruct commit *commit = NULL;\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349563","messageId":"0afa68a31d2b0dea1ea2d30f0452c31bbec8356f.1528317619.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1528317619.git.jonathantanmy@google.com","subject":"[PATCH v2 7/8] fetch-pack: introduce negotiator API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T20:47:13Z","receivedAt":"2018-06-06T20:47:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Introduce the new files fetch-negotiator.{h,c}, which contains an API\nbehind which the details of negotiation are abstracted. Currently, only\none algorithm is available: the existing one.\n\nThis patch is written to be easily reviewed: static functions are\nmoved verbatim from fetch-pack.c to negotiator/default.c, and it can be\nseen that the lines replaced by negotiator->X() calls are present in the\nX() functions respectively.\n\nSigned-off-by: Jonathan Tan \n---\n Makefile | 2 +\n fetch-negotiator.c | 8 ++\n fetch-negotiator.h | 57 ++++++++++++\n fetch-pack.c | 206 +++++++++----------------------------------\n negotiator/default.c | 176 ++++++++++++++++++++++++++++++++++++\n negotiator/default.h | 8 ++\n object.h | 3 +-\n 7 files changed, 297 insertions(+), 163 deletions(-)\n create mode 100644 fetch-negotiator.c\n create mode 100644 fetch-negotiator.h\n create mode 100644 negotiator/default.c\n create mode 100644 negotiator/default.h\n\ndiff --git a/Makefile b/Makefile\nindex 1d27f3636..96f84d1dc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -859,6 +859,7 @@ LIB_OBJS += ewah/ewah_bitmap.o\n LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec-cmd.o\n+LIB_OBJS += fetch-negotiator.o\n LIB_OBJS += fetch-object.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n@@ -891,6 +892,7 @@ LIB_OBJS += merge-blobs.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += mergesort.o\n LIB_OBJS += name-hash.o\n+LIB_OBJS += negotiator/default.o\n LIB_OBJS += notes.o\n LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\ndiff --git a/fetch-negotiator.c b/fetch-negotiator.c\nnew file mode 100644\nindex 000000000..2675d120f\n--- /dev/null\n+++ b/fetch-negotiator.c\n@@ -0,0 +1,8 @@\n+#include \"git-compat-util.h\"\n+#include \"fetch-negotiator.h\"\n+#include \"negotiator/default.h\"\n+\n+void fetch_negotiator_init(struct fetch_negotiator *negotiator)\n+{\n+\tdefault_negotiator_init(negotiator);\n+}\ndiff --git a/fetch-negotiator.h b/fetch-negotiator.h\nnew file mode 100644\nindex 000000000..b1290aa9c\n--- /dev/null\n+++ b/fetch-negotiator.h\n@@ -0,0 +1,57 @@\n+#ifndef FETCH_NEGOTIATOR\n+#define FETCH_NEGOTIATOR\n+\n+struct commit;\n+\n+/*\n+ * An object that supplies the information needed to negotiate the contents of\n+ * the to-be-sent packfile during a fetch.\n+ *\n+ * To set up the negotiator, call fetch_negotiator_init(), then known_common()\n+ * (0 or more times), then add_tip() (0 or more times).\n+ *\n+ * Then, when \"have\" lines are required, call next(). Call ack() to report what\n+ * the server tells us.\n+ *\n+ * Once negotiation is done, call release(). The negotiator then cannot be used\n+ * (unless reinitialized with fetch_negotiator_init()).\n+ */\n+struct fetch_negotiator {\n+\t/*\n+\t * Before negotiation starts, indicate that the server is known to have\n+\t * this commit.\n+\t */\n+\tvoid (*known_common)(struct fetch_negotiator *, struct commit *);\n+\n+\t/*\n+\t * Once this function is invoked, known_common() cannot be invoked any\n+\t * more.\n+\t *\n+\t * Indicate that this commit and all its ancestors are to be checked\n+\t * for commonality with the server.\n+\t */\n+\tvoid (*add_tip)(struct fetch_negotiator *, struct commit *);\n+\n+\t/*\n+\t * Once this function is invoked, known_common() and add_tip() cannot\n+\t * be invoked any more.\n+\t *\n+\t * Return the next commit that the client should send as a \"have\" line.\n+\t */\n+\tconst struct object_id *(*next)(struct fetch_negotiator *);\n+\n+\t/*\n+\t * Inform the negotiator that the server has the given commit. This\n+\t * method must only be called on commits returned by next().\n+\t */\n+\tint (*ack)(struct fetch_negotiator *, struct commit *);\n+\n+\tvoid (*release)(struct fetch_negotiator *);\n+\n+\t/* internal use */\n+\tvoid *data;\n+};\n+\n+void fetch_negotiator_init(struct fetch_negotiator *negotiator);\n+\n+#endif\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 4a4ec4da3..ad6f7ac32 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -15,10 +15,10 @@\n #include \"connect.h\"\n #include \"transport.h\"\n #include \"version.h\"\n-#include \"prio-queue.h\"\n #include \"sha1-array.h\"\n #include \"oidset.h\"\n #include \"packfile.h\"\n+#include \"fetch-negotiator.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -36,13 +36,7 @@ static const char *alternate_shallow_file;\n \n /* Remember to update object flag allocation in object.h */\n #define COMPLETE\t(1U << 0)\n-#define COMMON\t\t(1U << 1)\n-#define COMMON_REF\t(1U << 2)\n-#define SEEN\t\t(1U << 3)\n-#define POPPED\t\t(1U << 4)\n-#define ALTERNATE\t(1U << 5)\n-\n-static int marked;\n+#define ALTERNATE\t(1U << 1)\n \n /*\n * After sending this many \"have\"s if we do not get any new ACK , we\n@@ -50,11 +44,6 @@ static int marked;\n */\n #define MAX_IN_VAIN 256\n \n-struct data {\n-\tstruct prio_queue rev_list;\n-\tint non_common_revs;\n-};\n-\n static int multi_ack, use_sideband;\n /* Allow specifying sha1 if it is a ref tip. */\n #define ALLOW_TIP_SHA1\t01\n@@ -97,8 +86,8 @@ static void cache_one_alternate(const char *refname,\n \tcache->items[cache->nr++] = obj;\n }\n \n-static void for_each_cached_alternate(struct data *data,\n-\t\t\t\t void (*cb)(struct data *, struct object *))\n+static void for_each_cached_alternate(struct fetch_negotiator *negotiator,\n+\t\t\t\t void (*cb)(struct fetch_negotiator *, struct object *))\n {\n \tstatic int initialized;\n \tstatic struct alternate_object_cache cache;\n@@ -110,31 +99,17 @@ static void for_each_cached_alternate(struct data *data,\n \t}\n \n \tfor (i = 0; i < cache.nr; i++)\n-\t\tcb(data, cache.items[i]);\n-}\n-\n-static void rev_list_push(struct data *data, struct commit *commit, int mark)\n-{\n-\tif (!(commit->object.flags & mark)) {\n-\t\tcommit->object.flags |= mark;\n-\n-\t\tif (parse_commit(commit))\n-\t\t\treturn;\n-\n-\t\tprio_queue_put(&data->rev_list, commit);\n-\n-\t\tif (!(commit->object.flags & COMMON))\n-\t\t\tdata->non_common_revs++;\n-\t}\n+\t\tcb(negotiator, cache.items[i]);\n }\n \n-static int rev_list_insert_ref(struct data *data, const char *refname,\n+static int rev_list_insert_ref(struct fetch_negotiator *negotiator,\n+\t\t\t const char *refname,\n \t\t\t const struct object_id *oid)\n {\n \tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n \n \tif (o && o->type == OBJ_COMMIT)\n-\t\trev_list_push(data, (struct commit *)o, SEEN);\n+\t\tnegotiator->add_tip(negotiator, (struct commit *)o);\n \n \treturn 0;\n }\n@@ -145,97 +120,6 @@ static int rev_list_insert_ref_oid(const char *refname, const struct object_id *\n \treturn rev_list_insert_ref(cb_data, refname, oid);\n }\n \n-static int clear_marks(const char *refname, const struct object_id *oid,\n-\t\t int flag, void *cb_data)\n-{\n-\tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n-\n-\tif (o && o->type == OBJ_COMMIT)\n-\t\tclear_commit_marks((struct commit *)o,\n-\t\t\t\t COMMON | COMMON_REF | SEEN | POPPED);\n-\treturn 0;\n-}\n-\n-/*\n- This function marks a rev and its ancestors as common.\n- In some cases, it is desirable to mark only the ancestors (for example\n- when only the server does not yet know that they are common).\n-*/\n-\n-static void mark_common(struct data *data, struct commit *commit,\n-\t\tint ancestors_only, int dont_parse)\n-{\n-\tif (commit != NULL && !(commit->object.flags & COMMON)) {\n-\t\tstruct object *o = (struct object *)commit;\n-\n-\t\tif (!ancestors_only)\n-\t\t\to->flags |= COMMON;\n-\n-\t\tif (!(o->flags & SEEN))\n-\t\t\trev_list_push(data, commit, SEEN);\n-\t\telse {\n-\t\t\tstruct commit_list *parents;\n-\n-\t\t\tif (!ancestors_only && !(o->flags & POPPED))\n-\t\t\t\tdata->non_common_revs--;\n-\t\t\tif (!o->parsed && !dont_parse)\n-\t\t\t\tif (parse_commit(commit))\n-\t\t\t\t\treturn;\n-\n-\t\t\tfor (parents = commit->parents;\n-\t\t\t\t\tparents;\n-\t\t\t\t\tparents = parents->next)\n-\t\t\t\tmark_common(data, parents->item, 0, dont_parse);\n-\t\t}\n-\t}\n-}\n-\n-/*\n- Get the next rev to send, ignoring the common.\n-*/\n-\n-static const struct object_id *get_rev(struct data *data)\n-{\n-\tstruct commit *commit = NULL;\n-\n-\twhile (commit == NULL) {\n-\t\tunsigned int mark;\n-\t\tstruct commit_list *parents;\n-\n-\t\tif (data->rev_list.nr == 0 || data->non_common_revs == 0)\n-\t\t\treturn NULL;\n-\n-\t\tcommit = prio_queue_get(&data->rev_list);\n-\t\tparse_commit(commit);\n-\t\tparents = commit->parents;\n-\n-\t\tcommit->object.flags |= POPPED;\n-\t\tif (!(commit->object.flags & COMMON))\n-\t\t\tdata->non_common_revs--;\n-\n-\t\tif (commit->object.flags & COMMON) {\n-\t\t\t/* do not send \"have\", and ignore ancestors */\n-\t\t\tcommit = NULL;\n-\t\t\tmark = COMMON | SEEN;\n-\t\t} else if (commit->object.flags & COMMON_REF)\n-\t\t\t/* send \"have\", and ignore ancestors */\n-\t\t\tmark = COMMON | SEEN;\n-\t\telse\n-\t\t\t/* send \"have\", also for its ancestors */\n-\t\t\tmark = SEEN;\n-\n-\t\twhile (parents) {\n-\t\t\tif (!(parents->item->object.flags & SEEN))\n-\t\t\t\trev_list_push(data, parents->item, mark);\n-\t\t\tif (mark & COMMON)\n-\t\t\t\tmark_common(data, parents->item, 1, 0);\n-\t\t\tparents = parents->next;\n-\t\t}\n-\t}\n-\n-\treturn &commit->object.oid;\n-}\n-\n enum ack_type {\n \tNAK = 0,\n \tACK,\n@@ -302,9 +186,10 @@ static void send_request(struct fetch_pack_args *args,\n \t\twrite_or_die(fd, buf->buf, buf->len);\n }\n \n-static void insert_one_alternate_object(struct data *data, struct object *obj)\n+static void insert_one_alternate_object(struct fetch_negotiator *negotiator,\n+\t\t\t\t\tstruct object *obj)\n {\n-\trev_list_insert_ref(data, NULL, &obj->oid);\n+\trev_list_insert_ref(negotiator, NULL, &obj->oid);\n }\n \n #define INITIAL_FLUSH 16\n@@ -327,7 +212,8 @@ static int next_flush(int stateless_rpc, int count)\n \treturn count;\n }\n \n-static int find_common(struct data *data, struct fetch_pack_args *args,\n+static int find_common(struct fetch_negotiator *negotiator,\n+\t\t struct fetch_pack_args *args,\n \t\t int fd[2], struct object_id *result_oid,\n \t\t struct ref *refs)\n {\n@@ -343,8 +229,8 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \tif (args->stateless_rpc && multi_ack == 1)\n \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n \n-\tfor_each_ref(rev_list_insert_ref_oid, data);\n-\tfor_each_cached_alternate(data, insert_one_alternate_object);\n+\tfor_each_ref(rev_list_insert_ref_oid, negotiator);\n+\tfor_each_cached_alternate(negotiator, insert_one_alternate_object);\n \n \tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\n@@ -462,7 +348,7 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \tretval = -1;\n \tif (args->no_dependents)\n \t\tgoto done;\n-\twhile ((oid = get_rev(data))) {\n+\twhile ((oid = negotiator->next(negotiator))) {\n \t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tprint_verbose(args, \"have %s\", oid_to_hex(oid));\n \t\tin_vain++;\n@@ -502,8 +388,7 @@ static int find_common(struct data *data, struct fetch_pack_args *args,\n \t\t\t\t\tint was_common;\n \t\t\t\t\tif (!commit)\n \t\t\t\t\t\tdie(_(\"invalid commit %s\"), oid_to_hex(result_oid));\n-\t\t\t\t\twas_common = commit->object.flags & COMMON;\n-\t\t\t\t\tmark_common(data, commit, 0, 1);\n+\t\t\t\t\twas_common = negotiator->ack(negotiator, commit);\n \t\t\t\t\tif (args->stateless_rpc\n \t\t\t\t\t && ack == ACK_common\n \t\t\t\t\t && !was_common) {\n@@ -712,7 +597,8 @@ static void filter_refs(struct fetch_pack_args *args,\n \t*refs = newlist;\n }\n \n-static void mark_alternate_complete(struct data *unused, struct object *obj)\n+static void mark_alternate_complete(struct fetch_negotiator *unused,\n+\t\t\t\t struct object *obj)\n {\n \tmark_complete(&obj->oid);\n }\n@@ -749,7 +635,7 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n * earliest commit time of the objects in refs that are commits and that we know\n * the commit time of.\n */\n-static void mark_complete_and_common_ref(struct data *data,\n+static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n \t\t\t\t\t struct fetch_pack_args *args,\n \t\t\t\t\t struct ref **refs)\n {\n@@ -818,12 +704,8 @@ static void mark_complete_and_common_ref(struct data *data,\n \t\t\tif (!o || o->type != OBJ_COMMIT || !(o->flags & COMPLETE))\n \t\t\t\tcontinue;\n \n-\t\t\tif (!(o->flags & SEEN)) {\n-\t\t\t\trev_list_push(data, (struct commit *)o,\n-\t\t\t\t\t COMMON_REF | SEEN);\n-\n-\t\t\t\tmark_common(data, (struct commit *)o, 1, 1);\n-\t\t\t}\n+\t\t\tnegotiator->known_common(negotiator,\n+\t\t\t\t\t\t (struct commit *)o);\n \t\t}\n \t}\n \n@@ -1005,7 +887,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tstruct object_id oid;\n \tconst char *agent_feature;\n \tint agent_len;\n-\tstruct data data = { { compare_commits_by_commit_date } };\n+\tstruct fetch_negotiator negotiator;\n+\tfetch_negotiator_init(&negotiator);\n \n \tsort_ref_list(&ref, ref_compare_name);\n \tQSORT(sought, nr_sought, cmp_ref_by_name);\n@@ -1078,16 +961,13 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n \n-\tif (marked)\n-\t\tfor_each_ref(clear_marks, NULL);\n-\tmarked = 1;\n-\tmark_complete_and_common_ref(&data, args, &ref);\n+\tmark_complete_and_common_ref(&negotiator, args, &ref);\n \tfilter_refs(args, &ref, sought, nr_sought);\n \tif (everything_local(args, &ref)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n-\tif (find_common(&data, args, fd, &oid, ref) < 0)\n+\tif (find_common(&negotiator, args, fd, &oid, ref) < 0)\n \t\tif (!args->keep_pack)\n \t\t\t/* When cloning, it is not unusual to have\n \t\t\t * no common commit.\n@@ -1107,6 +987,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\tdie(_(\"git fetch-pack: fetch failed.\"));\n \n all_done:\n+\tnegotiator.release(&negotiator);\n \treturn ref;\n }\n \n@@ -1168,14 +1049,15 @@ static void add_common(struct strbuf *req_buf, struct oidset *common)\n \t}\n }\n \n-static int add_haves(struct data *data, struct strbuf *req_buf,\n+static int add_haves(struct fetch_negotiator *negotiator,\n+\t\t struct strbuf *req_buf,\n \t\t int *haves_to_send, int *in_vain)\n {\n \tint ret = 0;\n \tint haves_added = 0;\n \tconst struct object_id *oid;\n \n-\twhile ((oid = get_rev(data))) {\n+\twhile ((oid = negotiator->next(negotiator))) {\n \t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tif (++haves_added >= *haves_to_send)\n \t\t\tbreak;\n@@ -1194,7 +1076,7 @@ static int add_haves(struct data *data, struct strbuf *req_buf,\n \treturn ret;\n }\n \n-static int send_fetch_request(struct data *data, int fd_out,\n+static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t const struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain)\n@@ -1251,7 +1133,7 @@ static int send_fetch_request(struct data *data, int fd_out,\n \t\tadd_common(&req_buf, common);\n \n \t\t/* Add initial haves */\n-\t\tret = add_haves(data, &req_buf, haves_to_send, in_vain);\n+\t\tret = add_haves(negotiator, &req_buf, haves_to_send, in_vain);\n \t}\n \n \t/* Send request */\n@@ -1288,7 +1170,8 @@ static int process_section_header(struct packet_reader *reader,\n \treturn ret;\n }\n \n-static int process_acks(struct data *data, struct packet_reader *reader,\n+static int process_acks(struct fetch_negotiator *negotiator,\n+\t\t\tstruct packet_reader *reader,\n \t\t\tstruct oidset *common)\n {\n \t/* received */\n@@ -1308,7 +1191,7 @@ static int process_acks(struct data *data, struct packet_reader *reader,\n \t\t\t\tstruct commit *commit;\n \t\t\t\toidset_insert(common, &oid);\n \t\t\t\tcommit = lookup_commit(&oid);\n-\t\t\t\tmark_common(data, commit, 0, 1);\n+\t\t\t\tnegotiator->ack(negotiator, commit);\n \t\t\t}\n \t\t\tcontinue;\n \t\t}\n@@ -1386,7 +1269,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tstruct packet_reader reader;\n \tint in_vain = 0;\n \tint haves_to_send = INITIAL_FLUSH;\n-\tstruct data data = { { compare_commits_by_commit_date } };\n+\tstruct fetch_negotiator negotiator;\n+\tfetch_negotiator_init(&negotiator);\n \tpacket_reader_init(&reader, fd[0], NULL, 0,\n \t\t\t PACKET_READ_CHOMP_NEWLINE);\n \n@@ -1402,24 +1286,21 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n \t\t\t\targs->deepen = 1;\n \n-\t\t\tif (marked)\n-\t\t\t\tfor_each_ref(clear_marks, NULL);\n-\t\t\tmarked = 1;\n-\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n-\t\t\tmark_complete_and_common_ref(&data, args, &ref);\n+\t\t\tmark_complete_and_common_ref(&negotiator, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n \t\t\tif (everything_local(args, &ref))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n \n-\t\t\tfor_each_ref(rev_list_insert_ref_oid, &data);\n-\t\t\tfor_each_cached_alternate(&data,\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, &negotiator);\n+\t\t\tfor_each_cached_alternate(&negotiator,\n \t\t\t\t\t\t insert_one_alternate_object);\n \t\t\tbreak;\n \t\tcase FETCH_SEND_REQUEST:\n-\t\t\tif (send_fetch_request(&data, fd[1], args, ref, &common,\n+\t\t\tif (send_fetch_request(&negotiator, fd[1], args, ref,\n+\t\t\t\t\t &common,\n \t\t\t\t\t &haves_to_send, &in_vain))\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\telse\n@@ -1427,7 +1308,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tbreak;\n \t\tcase FETCH_PROCESS_ACKS:\n \t\t\t/* Process ACKs/NAKs */\n-\t\t\tswitch (process_acks(&data, &reader, &common)) {\n+\t\t\tswitch (process_acks(&negotiator, &reader, &common)) {\n \t\t\tcase 2:\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\t\tbreak;\n@@ -1456,6 +1337,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n+\tnegotiator.release(&negotiator);\n \toidset_clear(&common);\n \treturn ref;\n }\ndiff --git a/negotiator/default.c b/negotiator/default.c\nnew file mode 100644\nindex 000000000..b8f45cf78\n--- /dev/null\n+++ b/negotiator/default.c\n@@ -0,0 +1,176 @@\n+#include \"cache.h\"\n+#include \"default.h\"\n+#include \"../commit.h\"\n+#include \"../fetch-negotiator.h\"\n+#include \"../prio-queue.h\"\n+#include \"../refs.h\"\n+#include \"../tag.h\"\n+\n+/* Remember to update object flag allocation in object.h */\n+#define COMMON\t\t(1U << 2)\n+#define COMMON_REF\t(1U << 3)\n+#define SEEN\t\t(1U << 4)\n+#define POPPED\t\t(1U << 5)\n+\n+static int marked;\n+\n+struct data {\n+\tstruct prio_queue rev_list;\n+\tint non_common_revs;\n+};\n+\n+static void rev_list_push(struct data *data, struct commit *commit, int mark)\n+{\n+\tif (!(commit->object.flags & mark)) {\n+\t\tcommit->object.flags |= mark;\n+\n+\t\tif (parse_commit(commit))\n+\t\t\treturn;\n+\n+\t\tprio_queue_put(&data->rev_list, commit);\n+\n+\t\tif (!(commit->object.flags & COMMON))\n+\t\t\tdata->non_common_revs++;\n+\t}\n+}\n+\n+static int clear_marks(const char *refname, const struct object_id *oid,\n+\t\t int flag, void *cb_data)\n+{\n+\tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n+\n+\tif (o && o->type == OBJ_COMMIT)\n+\t\tclear_commit_marks((struct commit *)o,\n+\t\t\t\t COMMON | COMMON_REF | SEEN | POPPED);\n+\treturn 0;\n+}\n+\n+/*\n+ This function marks a rev and its ancestors as common.\n+ In some cases, it is desirable to mark only the ancestors (for example\n+ when only the server does not yet know that they are common).\n+*/\n+\n+static void mark_common(struct data *data, struct commit *commit,\n+\t\tint ancestors_only, int dont_parse)\n+{\n+\tif (commit != NULL && !(commit->object.flags & COMMON)) {\n+\t\tstruct object *o = (struct object *)commit;\n+\n+\t\tif (!ancestors_only)\n+\t\t\to->flags |= COMMON;\n+\n+\t\tif (!(o->flags & SEEN))\n+\t\t\trev_list_push(data, commit, SEEN);\n+\t\telse {\n+\t\t\tstruct commit_list *parents;\n+\n+\t\t\tif (!ancestors_only && !(o->flags & POPPED))\n+\t\t\t\tdata->non_common_revs--;\n+\t\t\tif (!o->parsed && !dont_parse)\n+\t\t\t\tif (parse_commit(commit))\n+\t\t\t\t\treturn;\n+\n+\t\t\tfor (parents = commit->parents;\n+\t\t\t\t\tparents;\n+\t\t\t\t\tparents = parents->next)\n+\t\t\t\tmark_common(data, parents->item, 0, dont_parse);\n+\t\t}\n+\t}\n+}\n+\n+/*\n+ Get the next rev to send, ignoring the common.\n+*/\n+\n+static const struct object_id *get_rev(struct data *data)\n+{\n+\tstruct commit *commit = NULL;\n+\n+\twhile (commit == NULL) {\n+\t\tunsigned int mark;\n+\t\tstruct commit_list *parents;\n+\n+\t\tif (data->rev_list.nr == 0 || data->non_common_revs == 0)\n+\t\t\treturn NULL;\n+\n+\t\tcommit = prio_queue_get(&data->rev_list);\n+\t\tparse_commit(commit);\n+\t\tparents = commit->parents;\n+\n+\t\tcommit->object.flags |= POPPED;\n+\t\tif (!(commit->object.flags & COMMON))\n+\t\t\tdata->non_common_revs--;\n+\n+\t\tif (commit->object.flags & COMMON) {\n+\t\t\t/* do not send \"have\", and ignore ancestors */\n+\t\t\tcommit = NULL;\n+\t\t\tmark = COMMON | SEEN;\n+\t\t} else if (commit->object.flags & COMMON_REF)\n+\t\t\t/* send \"have\", and ignore ancestors */\n+\t\t\tmark = COMMON | SEEN;\n+\t\telse\n+\t\t\t/* send \"have\", also for its ancestors */\n+\t\t\tmark = SEEN;\n+\n+\t\twhile (parents) {\n+\t\t\tif (!(parents->item->object.flags & SEEN))\n+\t\t\t\trev_list_push(data, parents->item, mark);\n+\t\t\tif (mark & COMMON)\n+\t\t\t\tmark_common(data, parents->item, 1, 0);\n+\t\t\tparents = parents->next;\n+\t\t}\n+\t}\n+\n+\treturn &commit->object.oid;\n+}\n+\n+static void known_common(struct fetch_negotiator *n, struct commit *c)\n+{\n+\tif (!(c->object.flags & SEEN)) {\n+\t\trev_list_push(n->data, c, COMMON_REF | SEEN);\n+\t\tmark_common(n->data, c, 1, 1);\n+\t}\n+}\n+\n+static void add_tip(struct fetch_negotiator *n, struct commit *c)\n+{\n+\tn->known_common = NULL;\n+\trev_list_push(n->data, c, SEEN);\n+}\n+\n+static const struct object_id *next(struct fetch_negotiator *n)\n+{\n+\tn->known_common = NULL;\n+\tn->add_tip = NULL;\n+\treturn get_rev(n->data);\n+}\n+\n+static int ack(struct fetch_negotiator *n, struct commit *c)\n+{\n+\tint known_to_be_common = !!(c->object.flags & COMMON);\n+\tmark_common(n->data, c, 0, 1);\n+\treturn known_to_be_common;\n+}\n+\n+static void release(struct fetch_negotiator *n)\n+{\n+\tclear_prio_queue(&((struct data *)n->data)->rev_list);\n+\tFREE_AND_NULL(n->data);\n+}\n+\n+void default_negotiator_init(struct fetch_negotiator *negotiator)\n+{\n+\tstruct data *data;\n+\tnegotiator->known_common = known_common;\n+\tnegotiator->add_tip = add_tip;\n+\tnegotiator->next = next;\n+\tnegotiator->ack = ack;\n+\tnegotiator->release = release;\n+\tnegotiator->data = data = xcalloc(1, sizeof(*data));\n+\tdata->rev_list.compare = compare_commits_by_commit_date;\n+\n+\tif (marked)\n+\t\tfor_each_ref(clear_marks, NULL);\n+\tmarked = 1;\n+}\ndiff --git a/negotiator/default.h b/negotiator/default.h\nnew file mode 100644\nindex 000000000..d23a8f2fb\n--- /dev/null\n+++ b/negotiator/default.h\n@@ -0,0 +1,8 @@\n+#ifndef NEGOTIATOR_DEFAULT_H\n+#define NEGOTIATOR_DEFAULT_H\n+\n+struct fetch_negotiator;\n+\n+void default_negotiator_init(struct fetch_negotiator *negotiator);\n+\n+#endif\ndiff --git a/object.h b/object.h\nindex 5c1395500..7db4941d6 100644\n--- a/object.h\n+++ b/object.h\n@@ -28,7 +28,8 @@ struct object_array {\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\n- * fetch-pack.c: 0----5\n+ * fetch-pack.c: 01\n+ * negotiator/default.c: 2--5\n * walker.c: 0-2\n * upload-pack.c: 4 11----------------19\n * builtin/blame.c: 12-13\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"349564","messageId":"fda04a33c38907388fd6d2166730928d78774c22.1528317619.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1528317619.git.jonathantanmy@google.com","subject":"[PATCH v2 3/8] fetch-pack: directly end negotiation if ACK ready","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-06T20:47:09Z","receivedAt":"2018-06-06T20:47:50Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"When \"ACK %s ready\" is received, find_common() clears rev_list in an\nattempt to stop further \"have\" lines from being sent [1]. It is much\nmore readable to explicitly break from the loop instead, so do this.\n\nThis means that the memory in priority queue will be reclaimed only upon\nprogram exit, similar to the cases in which \"ACK %s ready\" is not\nreceived. (A related problem occurs when do_fetch_pack() is invoked a\nsecond time in the same process with a possibly non-empty priority\nqueue, but this will be solved in a subsequent patch in this patch set.)\n\n[1] The rationale is further described in the originating commit\nf2cba9299b (\"fetch-pack: Finish negotation if remote replies \"ACK %s\nready\"\", 2011-03-14).\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 7 +++----\n 1 file changed, 3 insertions(+), 4 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 2812499a5..09f5c83c4 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -517,10 +517,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\t\t\tmark_common(commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgot_continue = 1;\n-\t\t\t\t\tif (ack == ACK_ready) {\n-\t\t\t\t\t\tclear_prio_queue(&rev_list);\n+\t\t\t\t\tif (ack == ACK_ready)\n \t\t\t\t\t\tgot_ready = 1;\n-\t\t\t\t\t}\n \t\t\t\t\tbreak;\n \t\t\t\t\t}\n \t\t\t\t}\n@@ -530,6 +528,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\t\tprint_verbose(args, _(\"giving up\"));\n \t\t\t\tbreak; /* give up */\n \t\t\t}\n+\t\t\tif (got_ready)\n+\t\t\t\tbreak;\n \t\t}\n \t}\n done:\n@@ -1300,7 +1300,6 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n \t\t}\n \n \t\tif (!strcmp(reader->line, \"ready\")) {\n-\t\t\tclear_prio_queue(&rev_list);\n \t\t\treceived_ready = 1;\n \t\t\tcontinue;\n \t\t}\n-- \n2.17.0.768.g1526ddbba1.dirty\n\n"},{"id":"350164","messageId":"20180614172615.GA220741@google.com","threadId":"48648","inReplyTo":"5687c0b22bf3b2bdbbe29b09788cc305a89710fc.1528317619.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 1/8] fetch-pack: split up everything_local()","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-06-14T17:26:15Z","receivedAt":"2018-06-14T17:26:21Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 06/06, Jonathan Tan wrote:\n> The function everything_local(), despite its name, also (1) marks\n> commits as COMPLETE and COMMON_REF and (2) invokes filter_refs() as\n> important side effects. Extract (1) into its own function\n> (mark_complete_and_common_ref()) and remove\n> (2).\n> \n> The restoring of save_commit_buffer, which was introduced in a1c6d7c1a7\n> (\"fetch-pack: restore save_commit_buffer after use\", 2017-12-08), is a\n> concern of the parse_object() call in mark_complete_and_common_ref(), so\n> it has been moved from the end of everything_local() to the end of\n> mark_complete_and_common_ref().\n\nThanks, this is much cleaner.\n\n> \n> Signed-off-by: Jonathan Tan \n> ---\n> fetch-pack.c | 39 ++++++++++++++++++++++++++++++---------\n> 1 file changed, 30 insertions(+), 9 deletions(-)\n> \n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index a320ce987..5c87bb8bb 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -734,12 +734,20 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n> \treturn 0;\n> }\n> \n> -static int everything_local(struct fetch_pack_args *args,\n> -\t\t\t struct ref **refs,\n> -\t\t\t struct ref **sought, int nr_sought)\n> +/*\n> + * Mark recent commits available locally and reachable from a local ref as\n> + * COMPLETE. If args->no_dependents is false, also mark COMPLETE remote refs as\n> + * COMMON_REF (otherwise, we are not planning to participate in negotiation, and\n> + * thus do not need COMMON_REF marks).\n> + *\n> + * The cutoff time for recency is determined by this heuristic: it is the\n> + * earliest commit time of the objects in refs that are commits and that we know\n> + * the commit time of.\n> + */\n> +static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n> +\t\t\t\t\t struct ref **refs)\n> {\n> \tstruct ref *ref;\n> -\tint retval;\n> \tint old_save_commit_buffer = save_commit_buffer;\n> \ttimestamp_t cutoff = 0;\n> \tstruct oidset loose_oid_set = OIDSET_INIT;\n> @@ -812,7 +820,18 @@ static int everything_local(struct fetch_pack_args *args,\n> \t\t}\n> \t}\n> \n> -\tfilter_refs(args, refs, sought, nr_sought);\n> +\tsave_commit_buffer = old_save_commit_buffer;\n> +}\n> +\n> +/*\n> + * Returns 1 if every object pointed to by the given remote refs is available\n> + * locally and reachable from a local ref, and 0 otherwise.\n> + */\n> +static int everything_local(struct fetch_pack_args *args,\n> +\t\t\t struct ref **refs)\n> +{\n> +\tstruct ref *ref;\n> +\tint retval;\n> \n> \tfor (retval = 1, ref = *refs; ref ; ref = ref->next) {\n> \t\tconst struct object_id *remote = &ref->old_oid;\n> @@ -829,8 +848,6 @@ static int everything_local(struct fetch_pack_args *args,\n> \t\t\t ref->name);\n> \t}\n> \n> -\tsave_commit_buffer = old_save_commit_buffer;\n> -\n> \treturn retval;\n> }\n> \n> @@ -1053,7 +1070,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n> \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n> \t\tdie(_(\"Server does not support --deepen\"));\n> \n> -\tif (everything_local(args, &ref, sought, nr_sought)) {\n> +\tmark_complete_and_common_ref(args, &ref);\n> +\tfilter_refs(args, &ref, sought, nr_sought);\n> +\tif (everything_local(args, &ref)) {\n> \t\tpacket_flush(fd[1]);\n> \t\tgoto all_done;\n> \t}\n> @@ -1377,7 +1396,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n> \n> \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> -\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n> +\t\t\tmark_complete_and_common_ref(args, &ref);\n> +\t\t\tfilter_refs(args, &ref, sought, nr_sought);\n> +\t\t\tif (everything_local(args, &ref))\n> \t\t\t\tstate = FETCH_DONE;\n> \t\t\telse\n> \t\t\t\tstate = FETCH_SEND_REQUEST;\n> -- \n> 2.17.0.768.g1526ddbba1.dirty\n> \n\n-- \nBrandon Williams\n"},{"id":"350165","messageId":"20180614172917.GB220741@google.com","threadId":"48648","inReplyTo":"fda04a33c38907388fd6d2166730928d78774c22.1528317619.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 3/8] fetch-pack: directly end negotiation if ACK ready","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-06-14T17:29:17Z","receivedAt":"2018-06-14T17:29:23Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 06/06, Jonathan Tan wrote:\n> When \"ACK %s ready\" is received, find_common() clears rev_list in an\n> attempt to stop further \"have\" lines from being sent [1]. It is much\n> more readable to explicitly break from the loop instead, so do this.\n> \n> This means that the memory in priority queue will be reclaimed only upon\n> program exit, similar to the cases in which \"ACK %s ready\" is not\n\nThis seems fine for now though ideally we would remove the global\npriority queue and have it live on the stack somewhere in the call stack\nand it could be cleared unconditionally before returning.\n\n> received. (A related problem occurs when do_fetch_pack() is invoked a\n> second time in the same process with a possibly non-empty priority\n> queue, but this will be solved in a subsequent patch in this patch set.)\n> \n> [1] The rationale is further described in the originating commit\n> f2cba9299b (\"fetch-pack: Finish negotation if remote replies \"ACK %s\n> ready\"\", 2011-03-14).\n> \n> Signed-off-by: Jonathan Tan \n> ---\n> fetch-pack.c | 7 +++----\n> 1 file changed, 3 insertions(+), 4 deletions(-)\n> \n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 2812499a5..09f5c83c4 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -517,10 +517,8 @@ static int find_common(struct fetch_pack_args *args,\n> \t\t\t\t\tmark_common(commit, 0, 1);\n> \t\t\t\t\tretval = 0;\n> \t\t\t\t\tgot_continue = 1;\n> -\t\t\t\t\tif (ack == ACK_ready) {\n> -\t\t\t\t\t\tclear_prio_queue(&rev_list);\n> +\t\t\t\t\tif (ack == ACK_ready)\n> \t\t\t\t\t\tgot_ready = 1;\n> -\t\t\t\t\t}\n> \t\t\t\t\tbreak;\n> \t\t\t\t\t}\n> \t\t\t\t}\n> @@ -530,6 +528,8 @@ static int find_common(struct fetch_pack_args *args,\n> \t\t\t\tprint_verbose(args, _(\"giving up\"));\n> \t\t\t\tbreak; /* give up */\n> \t\t\t}\n> +\t\t\tif (got_ready)\n> +\t\t\t\tbreak;\n> \t\t}\n> \t}\n> done:\n> @@ -1300,7 +1300,6 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n> \t\t}\n> \n> \t\tif (!strcmp(reader->line, \"ready\")) {\n> -\t\t\tclear_prio_queue(&rev_list);\n> \t\t\treceived_ready = 1;\n> \t\t\tcontinue;\n> \t\t}\n> -- \n> 2.17.0.768.g1526ddbba1.dirty\n> \n\n-- \nBrandon Williams\n"},{"id":"350167","messageId":"20180614173212.GC220741@google.com","threadId":"48648","inReplyTo":"f12342fb2760eb0449c86c66bf44d39f5871be57.1528317619.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 4/8] fetch-pack: use ref adv. to prune \"have\" sent","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-06-14T17:32:12Z","receivedAt":"2018-06-14T17:32:30Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 06/06, Jonathan Tan wrote:\n> In negotiation using protocol v2, fetch-pack sometimes does not make\n> full use of the information obtained in the ref advertisement:\n> specifically, that if the server advertises a commit that the client\n> also has, the client never needs to inform the server that it has the\n> commit's parents, since it can just tell the server that it has the\n> advertised commit and it knows that the server can and will infer the\n> rest.\n> \n> This is because, in do_fetch_pack_v2(), rev_list_insert_ref_oid() is\n> invoked before everything_local(). This means that if we have a commit\n> that is both our ref and their ref, it would be enqueued by\n> rev_list_insert_ref_oid() as SEEN, and since it is thus already SEEN,\n> everything_local() would not enqueue it.\n\nThanks for fixing this slight issue with v2. Though maybe we need to\nupdate the commit message here because a previous patch in this version\nof the series broke up everything_local() into various parts so that it\nis no longer responsible for enqueueing commits?\n\n> \n> If everything_local() were invoked first, as it is in do_fetch_pack()\n> for protocol v0, then everything_local() would enqueue it with\n> COMMON_REF | SEEN. The addition of COMMON_REF ensures that its parents\n> are not sent as \"have\" lines.\n> \n> Change the order in do_fetch_pack_v2() to be consistent with\n> do_fetch_pack(), and to avoid sending unnecessary \"have\" lines.\n> \n> Signed-off-by: Jonathan Tan \n> ---\n\n-- \nBrandon Williams\n"},{"id":"350168","messageId":"20180614173403.GD220741@google.com","threadId":"48648","inReplyTo":"20180614172917.GB220741@google.com","subject":"Re: [PATCH v2 3/8] fetch-pack: directly end negotiation if ACK ready","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-06-14T17:34:03Z","receivedAt":"2018-06-14T17:34:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 06/14, Brandon Williams wrote:\n> On 06/06, Jonathan Tan wrote:\n> > When \"ACK %s ready\" is received, find_common() clears rev_list in an\n> > attempt to stop further \"have\" lines from being sent [1]. It is much\n> > more readable to explicitly break from the loop instead, so do this.\n> > \n> > This means that the memory in priority queue will be reclaimed only upon\n> > program exit, similar to the cases in which \"ACK %s ready\" is not\n> \n> This seems fine for now though ideally we would remove the global\n> priority queue and have it live on the stack somewhere in the call stack\n> and it could be cleared unconditionally before returning.\n\nWait looks like a later commit does just this, maybe stick in a comment\nsaying a later patch is cleaning this up.\n\n> \n> > received. (A related problem occurs when do_fetch_pack() is invoked a\n> > second time in the same process with a possibly non-empty priority\n> > queue, but this will be solved in a subsequent patch in this patch set.)\n> > \n> > [1] The rationale is further described in the originating commit\n> > f2cba9299b (\"fetch-pack: Finish negotation if remote replies \"ACK %s\n> > ready\"\", 2011-03-14).\n> > \n> > Signed-off-by: Jonathan Tan \n> > ---\n> > fetch-pack.c | 7 +++----\n> > 1 file changed, 3 insertions(+), 4 deletions(-)\n> > \n> > diff --git a/fetch-pack.c b/fetch-pack.c\n> > index 2812499a5..09f5c83c4 100644\n> > --- a/fetch-pack.c\n> > +++ b/fetch-pack.c\n> > @@ -517,10 +517,8 @@ static int find_common(struct fetch_pack_args *args,\n> > \t\t\t\t\tmark_common(commit, 0, 1);\n> > \t\t\t\t\tretval = 0;\n> > \t\t\t\t\tgot_continue = 1;\n> > -\t\t\t\t\tif (ack == ACK_ready) {\n> > -\t\t\t\t\t\tclear_prio_queue(&rev_list);\n> > +\t\t\t\t\tif (ack == ACK_ready)\n> > \t\t\t\t\t\tgot_ready = 1;\n> > -\t\t\t\t\t}\n> > \t\t\t\t\tbreak;\n> > \t\t\t\t\t}\n> > \t\t\t\t}\n> > @@ -530,6 +528,8 @@ static int find_common(struct fetch_pack_args *args,\n> > \t\t\t\tprint_verbose(args, _(\"giving up\"));\n> > \t\t\t\tbreak; /* give up */\n> > \t\t\t}\n> > +\t\t\tif (got_ready)\n> > +\t\t\t\tbreak;\n> > \t\t}\n> > \t}\n> > done:\n> > @@ -1300,7 +1300,6 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n> > \t\t}\n> > \n> > \t\tif (!strcmp(reader->line, \"ready\")) {\n> > -\t\t\tclear_prio_queue(&rev_list);\n> > \t\t\treceived_ready = 1;\n> > \t\t\tcontinue;\n> > \t\t}\n> > -- \n> > 2.17.0.768.g1526ddbba1.dirty\n> > \n> \n> -- \n> Brandon Williams\n\n-- \nBrandon Williams\n"},{"id":"350172","messageId":"20180614173833.GE220741@google.com","threadId":"48648","inReplyTo":"955d9f62d2c8400791501d71c72fea4ef2dc1cff.1528317619.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 5/8] fetch-pack: make negotiation-related vars local","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-06-14T17:38:33Z","receivedAt":"2018-06-14T17:38:39Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 06/06, Jonathan Tan wrote:\n> Reduce the number of global variables by making the priority queue and\n> the count of non-common commits in it local, passing them as a struct to\n> various functions where necessary.\n> \n> This also helps in the case that fetch_pack() is invoked twice in the\n> same process (when tag following is required when using a transport that\n> does not support tag following), in that different priority queues will\n> now be used in each invocation, instead of reusing the possibly\n> non-empty one.\n> \n> The struct containing these variables is named \"data\" to ease review of\n> a subsequent patch in this series - in that patch, this struct\n> definition and several functions will be moved to a negotiation-specific\n> file, and this allows the move to be verbatim.\n\nOk I spoke too soon in my other mail. You've cleaned this up by moving\nthese things to local stack vars but you still don't clear the priority\nqueue meaning that memory is now leaking. I think you should insert a\ncall to clear the priority queue in the earlier patch before returning\nfrom the fetch code. This way you have a clear place to update that\ncall to clear in this patch so that you don't forget any memory leaks.\n\nI know this may still change later on in this series but it should still\nget taken care of to make the review process easier.\n\n> \n> Signed-off-by: Jonathan Tan \n> ---\n> fetch-pack.c | 104 +++++++++++++++++++++++++++++----------------------\n> 1 file changed, 59 insertions(+), 45 deletions(-)\n> \n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 114207b8e..c31644bb9 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -50,8 +50,12 @@ static int marked;\n> */\n> #define MAX_IN_VAIN 256\n> \n> -static struct prio_queue rev_list = { compare_commits_by_commit_date };\n> -static int non_common_revs, multi_ack, use_sideband;\n> +struct data {\n> +\tstruct prio_queue rev_list;\n> +\tint non_common_revs;\n> +};\n> +\n> +static int multi_ack, use_sideband;\n> /* Allow specifying sha1 if it is a ref tip. */\n> #define ALLOW_TIP_SHA1\t01\n> /* Allow request of a sha1 if it is reachable from a ref (possibly hidden ref). */\n> @@ -93,7 +97,8 @@ static void cache_one_alternate(const char *refname,\n> \tcache->items[cache->nr++] = obj;\n> }\n> \n> -static void for_each_cached_alternate(void (*cb)(struct object *))\n> +static void for_each_cached_alternate(struct data *data,\n> +\t\t\t\t void (*cb)(struct data *, struct object *))\n> {\n> \tstatic int initialized;\n> \tstatic struct alternate_object_cache cache;\n> @@ -105,10 +110,10 @@ static void for_each_cached_alternate(void (*cb)(struct object *))\n> \t}\n> \n> \tfor (i = 0; i < cache.nr; i++)\n> -\t\tcb(cache.items[i]);\n> +\t\tcb(data, cache.items[i]);\n> }\n> \n> -static void rev_list_push(struct commit *commit, int mark)\n> +static void rev_list_push(struct data *data, struct commit *commit, int mark)\n> {\n> \tif (!(commit->object.flags & mark)) {\n> \t\tcommit->object.flags |= mark;\n> @@ -116,19 +121,20 @@ static void rev_list_push(struct commit *commit, int mark)\n> \t\tif (parse_commit(commit))\n> \t\t\treturn;\n> \n> -\t\tprio_queue_put(&rev_list, commit);\n> +\t\tprio_queue_put(&data->rev_list, commit);\n> \n> \t\tif (!(commit->object.flags & COMMON))\n> -\t\t\tnon_common_revs++;\n> +\t\t\tdata->non_common_revs++;\n> \t}\n> }\n> \n> -static int rev_list_insert_ref(const char *refname, const struct object_id *oid)\n> +static int rev_list_insert_ref(struct data *data, const char *refname,\n> +\t\t\t const struct object_id *oid)\n> {\n> \tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n> \n> \tif (o && o->type == OBJ_COMMIT)\n> -\t\trev_list_push((struct commit *)o, SEEN);\n> +\t\trev_list_push(data, (struct commit *)o, SEEN);\n> \n> \treturn 0;\n> }\n> @@ -136,7 +142,7 @@ static int rev_list_insert_ref(const char *refname, const struct object_id *oid)\n> static int rev_list_insert_ref_oid(const char *refname, const struct object_id *oid,\n> \t\t\t\t int flag, void *cb_data)\n> {\n> -\treturn rev_list_insert_ref(refname, oid);\n> +\treturn rev_list_insert_ref(cb_data, refname, oid);\n> }\n> \n> static int clear_marks(const char *refname, const struct object_id *oid,\n> @@ -156,7 +162,7 @@ static int clear_marks(const char *refname, const struct object_id *oid,\n> when only the server does not yet know that they are common).\n> */\n> \n> -static void mark_common(struct commit *commit,\n> +static void mark_common(struct data *data, struct commit *commit,\n> \t\tint ancestors_only, int dont_parse)\n> {\n> \tif (commit != NULL && !(commit->object.flags & COMMON)) {\n> @@ -166,12 +172,12 @@ static void mark_common(struct commit *commit,\n> \t\t\to->flags |= COMMON;\n> \n> \t\tif (!(o->flags & SEEN))\n> -\t\t\trev_list_push(commit, SEEN);\n> +\t\t\trev_list_push(data, commit, SEEN);\n> \t\telse {\n> \t\t\tstruct commit_list *parents;\n> \n> \t\t\tif (!ancestors_only && !(o->flags & POPPED))\n> -\t\t\t\tnon_common_revs--;\n> +\t\t\t\tdata->non_common_revs--;\n> \t\t\tif (!o->parsed && !dont_parse)\n> \t\t\t\tif (parse_commit(commit))\n> \t\t\t\t\treturn;\n> @@ -179,7 +185,7 @@ static void mark_common(struct commit *commit,\n> \t\t\tfor (parents = commit->parents;\n> \t\t\t\t\tparents;\n> \t\t\t\t\tparents = parents->next)\n> -\t\t\t\tmark_common(parents->item, 0, dont_parse);\n> +\t\t\t\tmark_common(data, parents->item, 0, dont_parse);\n> \t\t}\n> \t}\n> }\n> @@ -188,7 +194,7 @@ static void mark_common(struct commit *commit,\n> Get the next rev to send, ignoring the common.\n> */\n> \n> -static const struct object_id *get_rev(void)\n> +static const struct object_id *get_rev(struct data *data)\n> {\n> \tstruct commit *commit = NULL;\n> \n> @@ -196,16 +202,16 @@ static const struct object_id *get_rev(void)\n> \t\tunsigned int mark;\n> \t\tstruct commit_list *parents;\n> \n> -\t\tif (rev_list.nr == 0 || non_common_revs == 0)\n> +\t\tif (data->rev_list.nr == 0 || data->non_common_revs == 0)\n> \t\t\treturn NULL;\n> \n> -\t\tcommit = prio_queue_get(&rev_list);\n> +\t\tcommit = prio_queue_get(&data->rev_list);\n> \t\tparse_commit(commit);\n> \t\tparents = commit->parents;\n> \n> \t\tcommit->object.flags |= POPPED;\n> \t\tif (!(commit->object.flags & COMMON))\n> -\t\t\tnon_common_revs--;\n> +\t\t\tdata->non_common_revs--;\n> \n> \t\tif (commit->object.flags & COMMON) {\n> \t\t\t/* do not send \"have\", and ignore ancestors */\n> @@ -220,9 +226,9 @@ static const struct object_id *get_rev(void)\n> \n> \t\twhile (parents) {\n> \t\t\tif (!(parents->item->object.flags & SEEN))\n> -\t\t\t\trev_list_push(parents->item, mark);\n> +\t\t\t\trev_list_push(data, parents->item, mark);\n> \t\t\tif (mark & COMMON)\n> -\t\t\t\tmark_common(parents->item, 1, 0);\n> +\t\t\t\tmark_common(data, parents->item, 1, 0);\n> \t\t\tparents = parents->next;\n> \t\t}\n> \t}\n> @@ -296,9 +302,9 @@ static void send_request(struct fetch_pack_args *args,\n> \t\twrite_or_die(fd, buf->buf, buf->len);\n> }\n> \n> -static void insert_one_alternate_object(struct object *obj)\n> +static void insert_one_alternate_object(struct data *data, struct object *obj)\n> {\n> -\trev_list_insert_ref(NULL, &obj->oid);\n> +\trev_list_insert_ref(data, NULL, &obj->oid);\n> }\n> \n> #define INITIAL_FLUSH 16\n> @@ -321,7 +327,7 @@ static int next_flush(int stateless_rpc, int count)\n> \treturn count;\n> }\n> \n> -static int find_common(struct fetch_pack_args *args,\n> +static int find_common(struct data *data, struct fetch_pack_args *args,\n> \t\t int fd[2], struct object_id *result_oid,\n> \t\t struct ref *refs)\n> {\n> @@ -337,8 +343,8 @@ static int find_common(struct fetch_pack_args *args,\n> \tif (args->stateless_rpc && multi_ack == 1)\n> \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n> \n> -\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n> -\tfor_each_cached_alternate(insert_one_alternate_object);\n> +\tfor_each_ref(rev_list_insert_ref_oid, data);\n> +\tfor_each_cached_alternate(data, insert_one_alternate_object);\n> \n> \tfetching = 0;\n> \tfor ( ; refs ; refs = refs->next) {\n> @@ -456,7 +462,7 @@ static int find_common(struct fetch_pack_args *args,\n> \tretval = -1;\n> \tif (args->no_dependents)\n> \t\tgoto done;\n> -\twhile ((oid = get_rev())) {\n> +\twhile ((oid = get_rev(data))) {\n> \t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n> \t\tprint_verbose(args, \"have %s\", oid_to_hex(oid));\n> \t\tin_vain++;\n> @@ -514,7 +520,7 @@ static int find_common(struct fetch_pack_args *args,\n> \t\t\t\t\t} else if (!args->stateless_rpc\n> \t\t\t\t\t\t || ack != ACK_common)\n> \t\t\t\t\t\tin_vain = 0;\n> -\t\t\t\t\tmark_common(commit, 0, 1);\n> +\t\t\t\t\tmark_common(data, commit, 0, 1);\n> \t\t\t\t\tretval = 0;\n> \t\t\t\t\tgot_continue = 1;\n> \t\t\t\t\tif (ack == ACK_ready)\n> @@ -704,7 +710,7 @@ static void filter_refs(struct fetch_pack_args *args,\n> \t*refs = newlist;\n> }\n> \n> -static void mark_alternate_complete(struct object *obj)\n> +static void mark_alternate_complete(struct data *unused, struct object *obj)\n> {\n> \tmark_complete(&obj->oid);\n> }\n> @@ -741,7 +747,8 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n> * earliest commit time of the objects in refs that are commits and that we know\n> * the commit time of.\n> */\n> -static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n> +static void mark_complete_and_common_ref(struct data *data,\n> +\t\t\t\t\t struct fetch_pack_args *args,\n> \t\t\t\t\t struct ref **refs)\n> {\n> \tstruct ref *ref;\n> @@ -792,7 +799,7 @@ static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n> \tif (!args->no_dependents) {\n> \t\tif (!args->deepen) {\n> \t\t\tfor_each_ref(mark_complete_oid, NULL);\n> -\t\t\tfor_each_cached_alternate(mark_alternate_complete);\n> +\t\t\tfor_each_cached_alternate(NULL, mark_alternate_complete);\n> \t\t\tcommit_list_sort_by_date(&complete);\n> \t\t\tif (cutoff)\n> \t\t\t\tmark_recent_complete_commits(args, cutoff);\n> @@ -810,9 +817,10 @@ static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n> \t\t\t\tcontinue;\n> \n> \t\t\tif (!(o->flags & SEEN)) {\n> -\t\t\t\trev_list_push((struct commit *)o, COMMON_REF | SEEN);\n> +\t\t\t\trev_list_push(data, (struct commit *)o,\n> +\t\t\t\t\t COMMON_REF | SEEN);\n> \n> -\t\t\t\tmark_common((struct commit *)o, 1, 1);\n> +\t\t\t\tmark_common(data, (struct commit *)o, 1, 1);\n> \t\t\t}\n> \t\t}\n> \t}\n> @@ -995,6 +1003,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n> \tstruct object_id oid;\n> \tconst char *agent_feature;\n> \tint agent_len;\n> +\tstruct data data = { { compare_commits_by_commit_date } };\n> \n> \tsort_ref_list(&ref, ref_compare_name);\n> \tQSORT(sought, nr_sought, cmp_ref_by_name);\n> @@ -1070,13 +1079,13 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n> \tif (marked)\n> \t\tfor_each_ref(clear_marks, NULL);\n> \tmarked = 1;\n> -\tmark_complete_and_common_ref(args, &ref);\n> +\tmark_complete_and_common_ref(&data, args, &ref);\n> \tfilter_refs(args, &ref, sought, nr_sought);\n> \tif (everything_local(args, &ref)) {\n> \t\tpacket_flush(fd[1]);\n> \t\tgoto all_done;\n> \t}\n> -\tif (find_common(args, fd, &oid, ref) < 0)\n> +\tif (find_common(&data, args, fd, &oid, ref) < 0)\n> \t\tif (!args->keep_pack)\n> \t\t\t/* When cloning, it is not unusual to have\n> \t\t\t * no common commit.\n> @@ -1157,13 +1166,14 @@ static void add_common(struct strbuf *req_buf, struct oidset *common)\n> \t}\n> }\n> \n> -static int add_haves(struct strbuf *req_buf, int *haves_to_send, int *in_vain)\n> +static int add_haves(struct data *data, struct strbuf *req_buf,\n> +\t\t int *haves_to_send, int *in_vain)\n> {\n> \tint ret = 0;\n> \tint haves_added = 0;\n> \tconst struct object_id *oid;\n> \n> -\twhile ((oid = get_rev())) {\n> +\twhile ((oid = get_rev(data))) {\n> \t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n> \t\tif (++haves_added >= *haves_to_send)\n> \t\t\tbreak;\n> @@ -1182,7 +1192,8 @@ static int add_haves(struct strbuf *req_buf, int *haves_to_send, int *in_vain)\n> \treturn ret;\n> }\n> \n> -static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n> +static int send_fetch_request(struct data *data, int fd_out,\n> +\t\t\t const struct fetch_pack_args *args,\n> \t\t\t const struct ref *wants, struct oidset *common,\n> \t\t\t int *haves_to_send, int *in_vain)\n> {\n> @@ -1238,7 +1249,7 @@ static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n> \t\tadd_common(&req_buf, common);\n> \n> \t\t/* Add initial haves */\n> -\t\tret = add_haves(&req_buf, haves_to_send, in_vain);\n> +\t\tret = add_haves(data, &req_buf, haves_to_send, in_vain);\n> \t}\n> \n> \t/* Send request */\n> @@ -1275,7 +1286,8 @@ static int process_section_header(struct packet_reader *reader,\n> \treturn ret;\n> }\n> \n> -static int process_acks(struct packet_reader *reader, struct oidset *common)\n> +static int process_acks(struct data *data, struct packet_reader *reader,\n> +\t\t\tstruct oidset *common)\n> {\n> \t/* received */\n> \tint received_ready = 0;\n> @@ -1294,7 +1306,7 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n> \t\t\t\tstruct commit *commit;\n> \t\t\t\toidset_insert(common, &oid);\n> \t\t\t\tcommit = lookup_commit(&oid);\n> -\t\t\t\tmark_common(commit, 0, 1);\n> +\t\t\t\tmark_common(data, commit, 0, 1);\n> \t\t\t}\n> \t\t\tcontinue;\n> \t\t}\n> @@ -1372,6 +1384,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \tstruct packet_reader reader;\n> \tint in_vain = 0;\n> \tint haves_to_send = INITIAL_FLUSH;\n> +\tstruct data data = { { compare_commits_by_commit_date } };\n> \tpacket_reader_init(&reader, fd[0], NULL, 0,\n> \t\t\t PACKET_READ_CHOMP_NEWLINE);\n> \n> @@ -1392,18 +1405,19 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\tmarked = 1;\n> \n> \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n> -\t\t\tmark_complete_and_common_ref(args, &ref);\n> +\t\t\tmark_complete_and_common_ref(&data, args, &ref);\n> \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n> \t\t\tif (everything_local(args, &ref))\n> \t\t\t\tstate = FETCH_DONE;\n> \t\t\telse\n> \t\t\t\tstate = FETCH_SEND_REQUEST;\n> \n> -\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n> -\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n> +\t\t\tfor_each_ref(rev_list_insert_ref_oid, &data);\n> +\t\t\tfor_each_cached_alternate(&data,\n> +\t\t\t\t\t\t insert_one_alternate_object);\n> \t\t\tbreak;\n> \t\tcase FETCH_SEND_REQUEST:\n> -\t\t\tif (send_fetch_request(fd[1], args, ref, &common,\n> +\t\t\tif (send_fetch_request(&data, fd[1], args, ref, &common,\n> \t\t\t\t\t &haves_to_send, &in_vain))\n> \t\t\t\tstate = FETCH_GET_PACK;\n> \t\t\telse\n> @@ -1411,7 +1425,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t\tbreak;\n> \t\tcase FETCH_PROCESS_ACKS:\n> \t\t\t/* Process ACKs/NAKs */\n> -\t\t\tswitch (process_acks(&reader, &common)) {\n> +\t\t\tswitch (process_acks(&data, &reader, &common)) {\n> \t\t\tcase 2:\n> \t\t\t\tstate = FETCH_GET_PACK;\n> \t\t\t\tbreak;\n> -- \n> 2.17.0.768.g1526ddbba1.dirty\n> \n\n-- \nBrandon Williams\n"},{"id":"350173","messageId":"20180614173947.GF220741@google.com","threadId":"48648","inReplyTo":"aa0df5f449c2f5c1f2e6d6edf933aad5f58b2a5e.1528317619.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 8/8] negotiator/default: use better style in comments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-06-14T17:39:47Z","receivedAt":"2018-06-14T17:39:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 06/06, Jonathan Tan wrote:\n> Signed-off-by: Jonathan Tan \n> ---\n> negotiator/default.c | 14 ++++++--------\n> 1 file changed, 6 insertions(+), 8 deletions(-)\n> \n> diff --git a/negotiator/default.c b/negotiator/default.c\n> index b8f45cf78..a9e52c943 100644\n> --- a/negotiator/default.c\n> +++ b/negotiator/default.c\n> @@ -46,11 +46,10 @@ static int clear_marks(const char *refname, const struct object_id *oid,\n> }\n> \n> /*\n> - This function marks a rev and its ancestors as common.\n> - In some cases, it is desirable to mark only the ancestors (for example\n> - when only the server does not yet know that they are common).\n> -*/\n> -\n> + * This function marks a rev and its ancestors as common.\n> + * In some cases, it is desirable to mark only the ancestors (for example\n> + * when only the server does not yet know that they are common).\n> + */\n> static void mark_common(struct data *data, struct commit *commit,\n> \t\tint ancestors_only, int dont_parse)\n> {\n> @@ -80,9 +79,8 @@ static void mark_common(struct data *data, struct commit *commit,\n> }\n> \n> /*\n> - Get the next rev to send, ignoring the common.\n> -*/\n> -\n> + * Get the next rev to send, ignoring the common.\n> + */\n> static const struct object_id *get_rev(struct data *data)\n> {\n> \tstruct commit *commit = NULL;\n> -- \n> 2.17.0.768.g1526ddbba1.dirty\n> \n\nDon't have this be a separate patch, squash it into the previous patch.\n\n-- \nBrandon Williams\n"},{"id":"350192","messageId":"xmqq36xp40uk.fsf@gitster-ct.c.googlers.com","threadId":"48648","inReplyTo":"955d9f62d2c8400791501d71c72fea4ef2dc1cff.1528317619.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 5/8] fetch-pack: make negotiation-related vars local","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-14T19:36:19Z","receivedAt":"2018-06-14T19:36:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> -static struct prio_queue rev_list = { compare_commits_by_commit_date };\n> -static int non_common_revs, multi_ack, use_sideband;\n> +struct data {\n> +\tstruct prio_queue rev_list;\n> +\tint non_common_revs;\n> +};\n> +\n> +static int multi_ack, use_sideband;\n\nAside from Brandon's comments, I think passing these throughout the\ncallchain instead of having them as static-globals is a step in the\nright direction. Could you, however, give the structure a bit more\ndescriptive name, than simply \"data\"? Anything the computer works\non is \"data\", and there must be something more specific than that\nabout this particular struct ;-)\n\n"},{"id":"350195","messageId":"xmqqsh5p2lja.fsf@gitster-ct.c.googlers.com","threadId":"48648","inReplyTo":"f12342fb2760eb0449c86c66bf44d39f5871be57.1528317619.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 4/8] fetch-pack: use ref adv. to prune \"have\" sent","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-14T19:52:25Z","receivedAt":"2018-06-14T19:52:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> +test_expect_success 'use ref advertisement to prune \"have\" lines sent' '\n> +\trm -rf server client &&\n> +\tgit init server &&\n> +\ttest_commit -C server both_have_1 &&\n> +\tgit -C server tag -d both_have_1 &&\n> +\ttest_commit -C server both_have_2 &&\n> +\n> +\t# In this test, the ref name that only the server has is a prefix of all\n> +\t# other refs. This is because in protocol v2, the client sends\n> +\t# \"ref-prefix\" to limit the ref advertisement. Naming the ref \"bo\" means\n> +\t# that \"ref-prefix refs/tags/bo*\" is sent, resulting in the client also\n> +\t# knowing about refs/tags/both_have_2, just as it would when it uses\n> +\t# protocol v0.\n\nI have a mixed feeling about this test.\n\nThe client wants to fetch \"bo\" and nothing else in this example.\nAnd refs \"both_have_*\", which have *nothing* to do with the ref the\nclient actually wants, is advertised merely because \"both_have_*\"\nbegins with \"bo\" with v2. But that is an implementation detail that\nwe do not necessarily want to cast in stone, isn't it? After all,\nin future iterations of the protocol, we may find it too excessive\nthat we have to advertise both_have_1..both_have_10000 when the\nclient asks for bo and change either the pattern matching rule of\nwhat \"ref-prefix refs/tags/bo\" matches, or the ref-prefix sent by\nthe client, and at that point, the expectation that both_have_2 is\nsent as \"have\" but both_have_2^1 is not may have to change, no?\n\n> +\tgit clone server client &&\n> +\ttest_commit -C server bo &&\n> +\ttest_commit -C client client_has &&\n> +\n> +\t# In both protocol v0 and v2, ensure that the parent of both_have_2 is\n> +\t# not sent as a \"have\" line. The client should know that the server has\n> +\t# both_have_2, so it only needs to inform the server that it has\n> +\t# both_have_2, and the server can infer the rest.\n> +\n> +\trm -f trace &&\n> +\tcp -r client clientv0 &&\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv0 \\\n> +\t\tfetch origin bo &&\n\n> +\tgrep \"have $(git -C client rev-parse client_has)\" trace &&\n> +\tgrep \"have $(git -C client rev-parse both_have_2)\" trace &&\n> +\t! grep \"have $(git -C client rev-parse both_have_2^)\" trace &&\n> +\n> +\trm -f trace &&\n> +\tcp -r client clientv2 &&\n> +\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv2 -c protocol.version=2 \\\n> +\t\tfetch origin bo &&\n> +\tgrep \"have $(git -C client rev-parse client_has)\" trace &&\n> +\tgrep \"have $(git -C client rev-parse both_have_2)\" trace &&\n> +\t! grep \"have $(git -C client rev-parse both_have_2^)\" trace\n> +'\n> +\n> test_expect_success 'filtering by size' '\n> \trm -rf server client &&\n> \ttest_create_repo server &&\n"},{"id":"350212","messageId":"cover.1529016350.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1527894919.git.jonathantanmy@google.com","subject":"[PATCH v3 0/7] Refactor fetch negotiation into its own API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-14T22:54:21Z","receivedAt":"2018-06-14T22:54:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks, Brandon and Junio, for your comments.\n\nUpdates since v2:\n - nothing new in patches 1 and 2\n - patch 3: now clear priority queue unconditionally once we are done\n with it (as requested by Brandon in a comment for a later patch)\n - patch 4: updated commit message to not mention everything_local() (as\n pointed out by Brandon), updated test to not rely on the fact that\n fetch-pack uses prefix matching (thanks, Junio, for the observation)\n - patch 5: used a more descriptive name (\"struct negotiation_state\")\n for the struct, instead of \"struct data\"\n - squashed patch 8 into patch 7; this means that the comments are not\n moved verbatim, but for the reviewer, verbatim-ness of comments is\n probably not that important anyway\n\nJonathan Tan (7):\n fetch-pack: split up everything_local()\n fetch-pack: clear marks before re-marking\n fetch-pack: directly end negotiation if ACK ready\n fetch-pack: use ref adv. to prune \"have\" sent\n fetch-pack: make negotiation-related vars local\n fetch-pack: move common check and marking together\n fetch-pack: introduce negotiator API\n\n Makefile | 2 +\n fetch-negotiator.c | 8 ++\n fetch-negotiator.h | 57 ++++++++++\n fetch-pack.c | 255 ++++++++++++++----------------------------\n negotiator/default.c | 176 +++++++++++++++++++++++++++++\n negotiator/default.h | 8 ++\n object.h | 3 +-\n t/t5500-fetch-pack.sh | 33 ++++++\n 8 files changed, 373 insertions(+), 169 deletions(-)\n create mode 100644 fetch-negotiator.c\n create mode 100644 fetch-negotiator.h\n create mode 100644 negotiator/default.c\n create mode 100644 negotiator/default.h\n\n-- \n2.17.0.582.gccdcbd54c4\n\n"},{"id":"350213","messageId":"5687c0b22bf3b2bdbbe29b09788cc305a89710fc.1529016350.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1529016350.git.jonathantanmy@google.com","subject":"[PATCH v3 1/7] fetch-pack: split up everything_local()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-14T22:54:22Z","receivedAt":"2018-06-14T22:54:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function everything_local(), despite its name, also (1) marks\ncommits as COMPLETE and COMMON_REF and (2) invokes filter_refs() as\nimportant side effects. Extract (1) into its own function\n(mark_complete_and_common_ref()) and remove\n(2).\n\nThe restoring of save_commit_buffer, which was introduced in a1c6d7c1a7\n(\"fetch-pack: restore save_commit_buffer after use\", 2017-12-08), is a\nconcern of the parse_object() call in mark_complete_and_common_ref(), so\nit has been moved from the end of everything_local() to the end of\nmark_complete_and_common_ref().\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 39 ++++++++++++++++++++++++++++++---------\n 1 file changed, 30 insertions(+), 9 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex a320ce987..5c87bb8bb 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -734,12 +734,20 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n \treturn 0;\n }\n \n-static int everything_local(struct fetch_pack_args *args,\n-\t\t\t struct ref **refs,\n-\t\t\t struct ref **sought, int nr_sought)\n+/*\n+ * Mark recent commits available locally and reachable from a local ref as\n+ * COMPLETE. If args->no_dependents is false, also mark COMPLETE remote refs as\n+ * COMMON_REF (otherwise, we are not planning to participate in negotiation, and\n+ * thus do not need COMMON_REF marks).\n+ *\n+ * The cutoff time for recency is determined by this heuristic: it is the\n+ * earliest commit time of the objects in refs that are commits and that we know\n+ * the commit time of.\n+ */\n+static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n+\t\t\t\t\t struct ref **refs)\n {\n \tstruct ref *ref;\n-\tint retval;\n \tint old_save_commit_buffer = save_commit_buffer;\n \ttimestamp_t cutoff = 0;\n \tstruct oidset loose_oid_set = OIDSET_INIT;\n@@ -812,7 +820,18 @@ static int everything_local(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n-\tfilter_refs(args, refs, sought, nr_sought);\n+\tsave_commit_buffer = old_save_commit_buffer;\n+}\n+\n+/*\n+ * Returns 1 if every object pointed to by the given remote refs is available\n+ * locally and reachable from a local ref, and 0 otherwise.\n+ */\n+static int everything_local(struct fetch_pack_args *args,\n+\t\t\t struct ref **refs)\n+{\n+\tstruct ref *ref;\n+\tint retval;\n \n \tfor (retval = 1, ref = *refs; ref ; ref = ref->next) {\n \t\tconst struct object_id *remote = &ref->old_oid;\n@@ -829,8 +848,6 @@ static int everything_local(struct fetch_pack_args *args,\n \t\t\t ref->name);\n \t}\n \n-\tsave_commit_buffer = old_save_commit_buffer;\n-\n \treturn retval;\n }\n \n@@ -1053,7 +1070,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n \n-\tif (everything_local(args, &ref, sought, nr_sought)) {\n+\tmark_complete_and_common_ref(args, &ref);\n+\tfilter_refs(args, &ref, sought, nr_sought);\n+\tif (everything_local(args, &ref)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n@@ -1377,7 +1396,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n \n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n-\t\t\tif (everything_local(args, &ref, sought, nr_sought))\n+\t\t\tmark_complete_and_common_ref(args, &ref);\n+\t\t\tfilter_refs(args, &ref, sought, nr_sought);\n+\t\t\tif (everything_local(args, &ref))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n-- \n2.17.0.582.gccdcbd54c4\n\n"},{"id":"350214","messageId":"dca6619c551b8259aef0476146184996657a80e9.1529016350.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1529016350.git.jonathantanmy@google.com","subject":"[PATCH v3 2/7] fetch-pack: clear marks before re-marking","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-14T22:54:23Z","receivedAt":"2018-06-14T22:54:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"If tag following is required when using a transport that does not\nsupport tag following, fetch_pack() will be invoked twice in the same\nprocess, necessitating a clearing of the object flags used by\nfetch_pack() sometime during the second invocation. This is currently\ndone in find_common(), which means that the invocation of\nmark_complete_and_common_ref() in do_fetch_pack() is useless.\n\n(This cannot be reproduced with Git alone, because all transports that\ncome with Git support tag following.)\n\nTherefore, move this clearing from find_common() to its\nparent function do_fetch_pack(), right before it calls\nmark_complete_and_common_ref().\n\nThis has been occurring since the commit that introduced the clearing of\nmarks, 420e9af498 (\"Fix tag following\", 2008-03-19).\n\nThe corresponding code for protocol v2 in do_fetch_pack_v2() does not\nhave this problem, as the clearing of flags is done before any marking\n(whether by rev_list_insert_ref_oid() or\nmark_complete_and_common_ref()).\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 5c87bb8bb..2812499a5 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -336,9 +336,6 @@ static int find_common(struct fetch_pack_args *args,\n \n \tif (args->stateless_rpc && multi_ack == 1)\n \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n-\tif (marked)\n-\t\tfor_each_ref(clear_marks, NULL);\n-\tmarked = 1;\n \n \tfor_each_ref(rev_list_insert_ref_oid, NULL);\n \tfor_each_cached_alternate(insert_one_alternate_object);\n@@ -1070,6 +1067,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n \n+\tif (marked)\n+\t\tfor_each_ref(clear_marks, NULL);\n+\tmarked = 1;\n \tmark_complete_and_common_ref(args, &ref);\n \tfilter_refs(args, &ref, sought, nr_sought);\n \tif (everything_local(args, &ref)) {\n-- \n2.17.0.582.gccdcbd54c4\n\n"},{"id":"350215","messageId":"fdcd88ba460d1feddb40037ff5b935339a51368b.1529016350.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1529016350.git.jonathantanmy@google.com","subject":"[PATCH v3 3/7] fetch-pack: directly end negotiation if ACK ready","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-14T22:54:24Z","receivedAt":"2018-06-14T22:54:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"When \"ACK %s ready\" is received, find_common() clears rev_list in an\nattempt to stop further \"have\" lines from being sent [1]. It is much\nmore readable to explicitly break from the loop instead.\n\nSo explicitly break from the loop, and make the clearing of the rev_list\nhappen unconditionally.\n\n[1] The rationale is further described in the originating commit\nf2cba9299b (\"fetch-pack: Finish negotation if remote replies \"ACK %s\nready\"\", 2011-03-14).\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 9 +++++----\n 1 file changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 2812499a5..60adfc073 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -517,10 +517,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\t\t\tmark_common(commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgot_continue = 1;\n-\t\t\t\t\tif (ack == ACK_ready) {\n-\t\t\t\t\t\tclear_prio_queue(&rev_list);\n+\t\t\t\t\tif (ack == ACK_ready)\n \t\t\t\t\t\tgot_ready = 1;\n-\t\t\t\t\t}\n \t\t\t\t\tbreak;\n \t\t\t\t\t}\n \t\t\t\t}\n@@ -530,6 +528,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\t\tprint_verbose(args, _(\"giving up\"));\n \t\t\t\tbreak; /* give up */\n \t\t\t}\n+\t\t\tif (got_ready)\n+\t\t\t\tbreak;\n \t\t}\n \t}\n done:\n@@ -1096,6 +1096,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\tdie(_(\"git fetch-pack: fetch failed.\"));\n \n all_done:\n+\tclear_prio_queue(&rev_list);\n \treturn ref;\n }\n \n@@ -1300,7 +1301,6 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n \t\t}\n \n \t\tif (!strcmp(reader->line, \"ready\")) {\n-\t\t\tclear_prio_queue(&rev_list);\n \t\t\treceived_ready = 1;\n \t\t\tcontinue;\n \t\t}\n@@ -1441,6 +1441,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n+\tclear_prio_queue(&rev_list);\n \toidset_clear(&common);\n \treturn ref;\n }\n-- \n2.17.0.582.gccdcbd54c4\n\n"},{"id":"350216","messageId":"be3774cd799b7ff538fefc2171bc46c164f2eae6.1529016350.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1529016350.git.jonathantanmy@google.com","subject":"[PATCH v3 4/7] fetch-pack: use ref adv. to prune \"have\" sent","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-14T22:54:25Z","receivedAt":"2018-06-14T22:54:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In negotiation using protocol v2, fetch-pack sometimes does not make\nfull use of the information obtained in the ref advertisement:\nspecifically, that if the server advertises a commit that the client\nalso has, the client never needs to inform the server that it has the\ncommit's parents, since it can just tell the server that it has the\nadvertised commit and it knows that the server can and will infer the\nrest.\n\nThis is because, in do_fetch_pack_v2(), rev_list_insert_ref_oid() is\ninvoked before mark_complete_and_common_ref(). This means that if we\nhave a commit that is both our ref and their ref, it would be enqueued\nby rev_list_insert_ref_oid() as SEEN, and since it is thus already SEEN,\nmark_complete_and_common_ref() would not enqueue it.\n\nIf mark_complete_and_common_ref() were invoked first, as it is in\ndo_fetch_pack() for protocol v0, then mark_complete_and_common_ref()\nwould enqueue it with COMMON_REF | SEEN. The addition of COMMON_REF\nensures that its parents are not sent as \"have\" lines.\n\nChange the order in do_fetch_pack_v2() to be consistent with\ndo_fetch_pack(), and to avoid sending unnecessary \"have\" lines.\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 6 +++---\n t/t5500-fetch-pack.sh | 33 +++++++++++++++++++++++++++++++++\n 2 files changed, 36 insertions(+), 3 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 60adfc073..806c40021 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -1392,9 +1392,6 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t\tfor_each_ref(clear_marks, NULL);\n \t\t\tmarked = 1;\n \n-\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n-\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n \t\t\tmark_complete_and_common_ref(args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n@@ -1402,6 +1399,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n+\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n+\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n \t\t\tbreak;\n \t\tcase FETCH_SEND_REQUEST:\n \t\t\tif (send_fetch_request(fd[1], args, ref, &common,\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex d4f435155..e0cdc295d 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -755,6 +755,39 @@ test_expect_success 'fetching deepen' '\n \t)\n '\n \n+test_expect_success 'use ref advertisement to prune \"have\" lines sent' '\n+\trm -rf server client &&\n+\tgit init server &&\n+\ttest_commit -C server both_have_1 &&\n+\tgit -C server tag -d both_have_1 &&\n+\ttest_commit -C server both_have_2 &&\n+\n+\tgit clone server client &&\n+\ttest_commit -C server server_has &&\n+\ttest_commit -C client client_has &&\n+\n+\t# In both protocol v0 and v2, ensure that the parent of both_have_2 is\n+\t# not sent as a \"have\" line. The client should know that the server has\n+\t# both_have_2, so it only needs to inform the server that it has\n+\t# both_have_2, and the server can infer the rest.\n+\n+\trm -f trace &&\n+\tcp -r client clientv0 &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv0 \\\n+\t\tfetch origin server_has both_have_2 &&\n+\tgrep \"have $(git -C client rev-parse client_has)\" trace &&\n+\tgrep \"have $(git -C client rev-parse both_have_2)\" trace &&\n+\t! grep \"have $(git -C client rev-parse both_have_2^)\" trace &&\n+\n+\trm -f trace &&\n+\tcp -r client clientv2 &&\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C clientv2 -c protocol.version=2 \\\n+\t\tfetch origin server_has both_have_2 &&\n+\tgrep \"have $(git -C client rev-parse client_has)\" trace &&\n+\tgrep \"have $(git -C client rev-parse both_have_2)\" trace &&\n+\t! grep \"have $(git -C client rev-parse both_have_2^)\" trace\n+'\n+\n test_expect_success 'filtering by size' '\n \trm -rf server client &&\n \ttest_create_repo server &&\n-- \n2.17.0.582.gccdcbd54c4\n\n"},{"id":"350217","messageId":"b868755fb9645194be672a03532170f125169944.1529016350.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1529016350.git.jonathantanmy@google.com","subject":"[PATCH v3 5/7] fetch-pack: make negotiation-related vars local","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-14T22:54:26Z","receivedAt":"2018-06-14T22:54:48Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Reduce the number of global variables by making the priority queue and\nthe count of non-common commits in it local, passing them as a struct to\nvarious functions where necessary.\n\nThis also helps in the case that fetch_pack() is invoked twice in the\nsame process (when tag following is required when using a transport that\ndoes not support tag following), in that different priority queues will\nnow be used in each invocation, instead of reusing the possibly\nnon-empty one.\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 116 ++++++++++++++++++++++++++++++---------------------\n 1 file changed, 69 insertions(+), 47 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 806c40021..473e415c5 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -50,8 +50,12 @@ static int marked;\n */\n #define MAX_IN_VAIN 256\n \n-static struct prio_queue rev_list = { compare_commits_by_commit_date };\n-static int non_common_revs, multi_ack, use_sideband;\n+struct negotiation_state {\n+\tstruct prio_queue rev_list;\n+\tint non_common_revs;\n+};\n+\n+static int multi_ack, use_sideband;\n /* Allow specifying sha1 if it is a ref tip. */\n #define ALLOW_TIP_SHA1\t01\n /* Allow request of a sha1 if it is reachable from a ref (possibly hidden ref). */\n@@ -93,7 +97,9 @@ static void cache_one_alternate(const char *refname,\n \tcache->items[cache->nr++] = obj;\n }\n \n-static void for_each_cached_alternate(void (*cb)(struct object *))\n+static void for_each_cached_alternate(struct negotiation_state *ns,\n+\t\t\t\t void (*cb)(struct negotiation_state *,\n+\t\t\t\t\t struct object *))\n {\n \tstatic int initialized;\n \tstatic struct alternate_object_cache cache;\n@@ -105,10 +111,11 @@ static void for_each_cached_alternate(void (*cb)(struct object *))\n \t}\n \n \tfor (i = 0; i < cache.nr; i++)\n-\t\tcb(cache.items[i]);\n+\t\tcb(ns, cache.items[i]);\n }\n \n-static void rev_list_push(struct commit *commit, int mark)\n+static void rev_list_push(struct negotiation_state *ns,\n+\t\t\t struct commit *commit, int mark)\n {\n \tif (!(commit->object.flags & mark)) {\n \t\tcommit->object.flags |= mark;\n@@ -116,19 +123,21 @@ static void rev_list_push(struct commit *commit, int mark)\n \t\tif (parse_commit(commit))\n \t\t\treturn;\n \n-\t\tprio_queue_put(&rev_list, commit);\n+\t\tprio_queue_put(&ns->rev_list, commit);\n \n \t\tif (!(commit->object.flags & COMMON))\n-\t\t\tnon_common_revs++;\n+\t\t\tns->non_common_revs++;\n \t}\n }\n \n-static int rev_list_insert_ref(const char *refname, const struct object_id *oid)\n+static int rev_list_insert_ref(struct negotiation_state *ns,\n+\t\t\t const char *refname,\n+\t\t\t const struct object_id *oid)\n {\n \tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n \n \tif (o && o->type == OBJ_COMMIT)\n-\t\trev_list_push((struct commit *)o, SEEN);\n+\t\trev_list_push(ns, (struct commit *)o, SEEN);\n \n \treturn 0;\n }\n@@ -136,7 +145,7 @@ static int rev_list_insert_ref(const char *refname, const struct object_id *oid)\n static int rev_list_insert_ref_oid(const char *refname, const struct object_id *oid,\n \t\t\t\t int flag, void *cb_data)\n {\n-\treturn rev_list_insert_ref(refname, oid);\n+\treturn rev_list_insert_ref(cb_data, refname, oid);\n }\n \n static int clear_marks(const char *refname, const struct object_id *oid,\n@@ -156,7 +165,7 @@ static int clear_marks(const char *refname, const struct object_id *oid,\n when only the server does not yet know that they are common).\n */\n \n-static void mark_common(struct commit *commit,\n+static void mark_common(struct negotiation_state *ns, struct commit *commit,\n \t\tint ancestors_only, int dont_parse)\n {\n \tif (commit != NULL && !(commit->object.flags & COMMON)) {\n@@ -166,12 +175,12 @@ static void mark_common(struct commit *commit,\n \t\t\to->flags |= COMMON;\n \n \t\tif (!(o->flags & SEEN))\n-\t\t\trev_list_push(commit, SEEN);\n+\t\t\trev_list_push(ns, commit, SEEN);\n \t\telse {\n \t\t\tstruct commit_list *parents;\n \n \t\t\tif (!ancestors_only && !(o->flags & POPPED))\n-\t\t\t\tnon_common_revs--;\n+\t\t\t\tns->non_common_revs--;\n \t\t\tif (!o->parsed && !dont_parse)\n \t\t\t\tif (parse_commit(commit))\n \t\t\t\t\treturn;\n@@ -179,7 +188,8 @@ static void mark_common(struct commit *commit,\n \t\t\tfor (parents = commit->parents;\n \t\t\t\t\tparents;\n \t\t\t\t\tparents = parents->next)\n-\t\t\t\tmark_common(parents->item, 0, dont_parse);\n+\t\t\t\tmark_common(ns, parents->item, 0,\n+\t\t\t\t\t dont_parse);\n \t\t}\n \t}\n }\n@@ -188,7 +198,7 @@ static void mark_common(struct commit *commit,\n Get the next rev to send, ignoring the common.\n */\n \n-static const struct object_id *get_rev(void)\n+static const struct object_id *get_rev(struct negotiation_state *ns)\n {\n \tstruct commit *commit = NULL;\n \n@@ -196,16 +206,16 @@ static const struct object_id *get_rev(void)\n \t\tunsigned int mark;\n \t\tstruct commit_list *parents;\n \n-\t\tif (rev_list.nr == 0 || non_common_revs == 0)\n+\t\tif (ns->rev_list.nr == 0 || ns->non_common_revs == 0)\n \t\t\treturn NULL;\n \n-\t\tcommit = prio_queue_get(&rev_list);\n+\t\tcommit = prio_queue_get(&ns->rev_list);\n \t\tparse_commit(commit);\n \t\tparents = commit->parents;\n \n \t\tcommit->object.flags |= POPPED;\n \t\tif (!(commit->object.flags & COMMON))\n-\t\t\tnon_common_revs--;\n+\t\t\tns->non_common_revs--;\n \n \t\tif (commit->object.flags & COMMON) {\n \t\t\t/* do not send \"have\", and ignore ancestors */\n@@ -220,9 +230,9 @@ static const struct object_id *get_rev(void)\n \n \t\twhile (parents) {\n \t\t\tif (!(parents->item->object.flags & SEEN))\n-\t\t\t\trev_list_push(parents->item, mark);\n+\t\t\t\trev_list_push(ns, parents->item, mark);\n \t\t\tif (mark & COMMON)\n-\t\t\t\tmark_common(parents->item, 1, 0);\n+\t\t\t\tmark_common(ns, parents->item, 1, 0);\n \t\t\tparents = parents->next;\n \t\t}\n \t}\n@@ -296,9 +306,10 @@ static void send_request(struct fetch_pack_args *args,\n \t\twrite_or_die(fd, buf->buf, buf->len);\n }\n \n-static void insert_one_alternate_object(struct object *obj)\n+static void insert_one_alternate_object(struct negotiation_state *ns,\n+\t\t\t\t\tstruct object *obj)\n {\n-\trev_list_insert_ref(NULL, &obj->oid);\n+\trev_list_insert_ref(ns, NULL, &obj->oid);\n }\n \n #define INITIAL_FLUSH 16\n@@ -321,7 +332,8 @@ static int next_flush(int stateless_rpc, int count)\n \treturn count;\n }\n \n-static int find_common(struct fetch_pack_args *args,\n+static int find_common(struct negotiation_state *ns,\n+\t\t struct fetch_pack_args *args,\n \t\t int fd[2], struct object_id *result_oid,\n \t\t struct ref *refs)\n {\n@@ -337,8 +349,8 @@ static int find_common(struct fetch_pack_args *args,\n \tif (args->stateless_rpc && multi_ack == 1)\n \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n \n-\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\tfor_each_cached_alternate(insert_one_alternate_object);\n+\tfor_each_ref(rev_list_insert_ref_oid, ns);\n+\tfor_each_cached_alternate(ns, insert_one_alternate_object);\n \n \tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\n@@ -456,7 +468,7 @@ static int find_common(struct fetch_pack_args *args,\n \tretval = -1;\n \tif (args->no_dependents)\n \t\tgoto done;\n-\twhile ((oid = get_rev())) {\n+\twhile ((oid = get_rev(ns))) {\n \t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tprint_verbose(args, \"have %s\", oid_to_hex(oid));\n \t\tin_vain++;\n@@ -514,7 +526,7 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\t\t\t} else if (!args->stateless_rpc\n \t\t\t\t\t\t || ack != ACK_common)\n \t\t\t\t\t\tin_vain = 0;\n-\t\t\t\t\tmark_common(commit, 0, 1);\n+\t\t\t\t\tmark_common(ns, commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgot_continue = 1;\n \t\t\t\t\tif (ack == ACK_ready)\n@@ -704,7 +716,8 @@ static void filter_refs(struct fetch_pack_args *args,\n \t*refs = newlist;\n }\n \n-static void mark_alternate_complete(struct object *obj)\n+static void mark_alternate_complete(struct negotiation_state *unused,\n+\t\t\t\t struct object *obj)\n {\n \tmark_complete(&obj->oid);\n }\n@@ -741,7 +754,8 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n * earliest commit time of the objects in refs that are commits and that we know\n * the commit time of.\n */\n-static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n+static void mark_complete_and_common_ref(struct negotiation_state *ns,\n+\t\t\t\t\t struct fetch_pack_args *args,\n \t\t\t\t\t struct ref **refs)\n {\n \tstruct ref *ref;\n@@ -792,7 +806,7 @@ static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n \tif (!args->no_dependents) {\n \t\tif (!args->deepen) {\n \t\t\tfor_each_ref(mark_complete_oid, NULL);\n-\t\t\tfor_each_cached_alternate(mark_alternate_complete);\n+\t\t\tfor_each_cached_alternate(NULL, mark_alternate_complete);\n \t\t\tcommit_list_sort_by_date(&complete);\n \t\t\tif (cutoff)\n \t\t\t\tmark_recent_complete_commits(args, cutoff);\n@@ -810,9 +824,10 @@ static void mark_complete_and_common_ref(struct fetch_pack_args *args,\n \t\t\t\tcontinue;\n \n \t\t\tif (!(o->flags & SEEN)) {\n-\t\t\t\trev_list_push((struct commit *)o, COMMON_REF | SEEN);\n+\t\t\t\trev_list_push(ns, (struct commit *)o,\n+\t\t\t\t\t COMMON_REF | SEEN);\n \n-\t\t\t\tmark_common((struct commit *)o, 1, 1);\n+\t\t\t\tmark_common(ns, (struct commit *)o, 1, 1);\n \t\t\t}\n \t\t}\n \t}\n@@ -995,6 +1010,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tstruct object_id oid;\n \tconst char *agent_feature;\n \tint agent_len;\n+\tstruct negotiation_state ns = { { compare_commits_by_commit_date } };\n \n \tsort_ref_list(&ref, ref_compare_name);\n \tQSORT(sought, nr_sought, cmp_ref_by_name);\n@@ -1070,13 +1086,13 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (marked)\n \t\tfor_each_ref(clear_marks, NULL);\n \tmarked = 1;\n-\tmark_complete_and_common_ref(args, &ref);\n+\tmark_complete_and_common_ref(&ns, args, &ref);\n \tfilter_refs(args, &ref, sought, nr_sought);\n \tif (everything_local(args, &ref)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n-\tif (find_common(args, fd, &oid, ref) < 0)\n+\tif (find_common(&ns, args, fd, &oid, ref) < 0)\n \t\tif (!args->keep_pack)\n \t\t\t/* When cloning, it is not unusual to have\n \t\t\t * no common commit.\n@@ -1096,7 +1112,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\tdie(_(\"git fetch-pack: fetch failed.\"));\n \n all_done:\n-\tclear_prio_queue(&rev_list);\n+\tclear_prio_queue(&ns.rev_list);\n \treturn ref;\n }\n \n@@ -1158,13 +1174,14 @@ static void add_common(struct strbuf *req_buf, struct oidset *common)\n \t}\n }\n \n-static int add_haves(struct strbuf *req_buf, int *haves_to_send, int *in_vain)\n+static int add_haves(struct negotiation_state *ns, struct strbuf *req_buf,\n+\t\t int *haves_to_send, int *in_vain)\n {\n \tint ret = 0;\n \tint haves_added = 0;\n \tconst struct object_id *oid;\n \n-\twhile ((oid = get_rev())) {\n+\twhile ((oid = get_rev(ns))) {\n \t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tif (++haves_added >= *haves_to_send)\n \t\t\tbreak;\n@@ -1183,7 +1200,8 @@ static int add_haves(struct strbuf *req_buf, int *haves_to_send, int *in_vain)\n \treturn ret;\n }\n \n-static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n+static int send_fetch_request(struct negotiation_state *ns, int fd_out,\n+\t\t\t const struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain)\n {\n@@ -1239,7 +1257,7 @@ static int send_fetch_request(int fd_out, const struct fetch_pack_args *args,\n \t\tadd_common(&req_buf, common);\n \n \t\t/* Add initial haves */\n-\t\tret = add_haves(&req_buf, haves_to_send, in_vain);\n+\t\tret = add_haves(ns, &req_buf, haves_to_send, in_vain);\n \t}\n \n \t/* Send request */\n@@ -1276,7 +1294,9 @@ static int process_section_header(struct packet_reader *reader,\n \treturn ret;\n }\n \n-static int process_acks(struct packet_reader *reader, struct oidset *common)\n+static int process_acks(struct negotiation_state *ns,\n+\t\t\tstruct packet_reader *reader,\n+\t\t\tstruct oidset *common)\n {\n \t/* received */\n \tint received_ready = 0;\n@@ -1295,7 +1315,7 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n \t\t\t\tstruct commit *commit;\n \t\t\t\toidset_insert(common, &oid);\n \t\t\t\tcommit = lookup_commit(&oid);\n-\t\t\t\tmark_common(commit, 0, 1);\n+\t\t\t\tmark_common(ns, commit, 0, 1);\n \t\t\t}\n \t\t\tcontinue;\n \t\t}\n@@ -1373,6 +1393,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tstruct packet_reader reader;\n \tint in_vain = 0;\n \tint haves_to_send = INITIAL_FLUSH;\n+\tstruct negotiation_state ns = { { compare_commits_by_commit_date } };\n \tpacket_reader_init(&reader, fd[0], NULL, 0,\n \t\t\t PACKET_READ_CHOMP_NEWLINE);\n \n@@ -1393,18 +1414,19 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tmarked = 1;\n \n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n-\t\t\tmark_complete_and_common_ref(args, &ref);\n+\t\t\tmark_complete_and_common_ref(&ns, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n \t\t\tif (everything_local(args, &ref))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n \n-\t\t\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\t\t\tfor_each_cached_alternate(insert_one_alternate_object);\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, &ns);\n+\t\t\tfor_each_cached_alternate(&ns,\n+\t\t\t\t\t\t insert_one_alternate_object);\n \t\t\tbreak;\n \t\tcase FETCH_SEND_REQUEST:\n-\t\t\tif (send_fetch_request(fd[1], args, ref, &common,\n+\t\t\tif (send_fetch_request(&ns, fd[1], args, ref, &common,\n \t\t\t\t\t &haves_to_send, &in_vain))\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\telse\n@@ -1412,7 +1434,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tbreak;\n \t\tcase FETCH_PROCESS_ACKS:\n \t\t\t/* Process ACKs/NAKs */\n-\t\t\tswitch (process_acks(&reader, &common)) {\n+\t\t\tswitch (process_acks(&ns, &reader, &common)) {\n \t\t\tcase 2:\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\t\tbreak;\n@@ -1441,7 +1463,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n-\tclear_prio_queue(&rev_list);\n+\tclear_prio_queue(&ns.rev_list);\n \toidset_clear(&common);\n \treturn ref;\n }\n-- \n2.17.0.582.gccdcbd54c4\n\n"},{"id":"350218","messageId":"97f9a133e409f0fa898132a6f972e564db522dbf.1529016350.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1529016350.git.jonathantanmy@google.com","subject":"[PATCH v3 6/7] fetch-pack: move common check and marking together","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-14T22:54:27Z","receivedAt":"2018-06-14T22:54:53Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"When receiving 'ACK continue' for a common commit, check if\nthe commit was already known to be common and mark it as such if not up\nfront. This should make future refactoring of how the information about\ncommon commits is stored more straightforward.\n\nNo visible change intended.\n\nSigned-off-by: Jonathan Tan \n---\n fetch-pack.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 473e415c5..fb76d4017 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -505,11 +505,14 @@ static int find_common(struct negotiation_state *ns,\n \t\t\t\tcase ACK_continue: {\n \t\t\t\t\tstruct commit *commit =\n \t\t\t\t\t\tlookup_commit(result_oid);\n+\t\t\t\t\tint was_common;\n \t\t\t\t\tif (!commit)\n \t\t\t\t\t\tdie(_(\"invalid commit %s\"), oid_to_hex(result_oid));\n+\t\t\t\t\twas_common = commit->object.flags & COMMON;\n+\t\t\t\t\tmark_common(ns, commit, 0, 1);\n \t\t\t\t\tif (args->stateless_rpc\n \t\t\t\t\t && ack == ACK_common\n-\t\t\t\t\t && !(commit->object.flags & COMMON)) {\n+\t\t\t\t\t && !was_common) {\n \t\t\t\t\t\t/* We need to replay the have for this object\n \t\t\t\t\t\t * on the next RPC request so the peer knows\n \t\t\t\t\t\t * it is in common with us.\n@@ -526,7 +529,6 @@ static int find_common(struct negotiation_state *ns,\n \t\t\t\t\t} else if (!args->stateless_rpc\n \t\t\t\t\t\t || ack != ACK_common)\n \t\t\t\t\t\tin_vain = 0;\n-\t\t\t\t\tmark_common(ns, commit, 0, 1);\n \t\t\t\t\tretval = 0;\n \t\t\t\t\tgot_continue = 1;\n \t\t\t\t\tif (ack == ACK_ready)\n-- \n2.17.0.582.gccdcbd54c4\n\n"},{"id":"350219","messageId":"122fdd0266b139cac9be01c62592a370c037ffec.1529016350.git.jonathantanmy@google.com","threadId":"48648","inReplyTo":"cover.1529016350.git.jonathantanmy@google.com","subject":"[PATCH v3 7/7] fetch-pack: introduce negotiator API","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-14T22:54:28Z","receivedAt":"2018-06-14T22:54:56Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Introduce the new files fetch-negotiator.{h,c}, which contains an API\nbehind which the details of negotiation are abstracted. Currently, only\none algorithm is available: the existing one.\n\nThis patch is written to be easily reviewed: static functions are\nmoved verbatim from fetch-pack.c to negotiator/default.c, and it can be\nseen that the lines replaced by negotiator->X() calls are present in the\nX() functions respectively.\n\nSigned-off-by: Jonathan Tan \n---\n Makefile | 2 +\n fetch-negotiator.c | 8 ++\n fetch-negotiator.h | 57 ++++++++++++\n fetch-pack.c | 205 ++++++++-----------------------------------\n negotiator/default.c | 176 +++++++++++++++++++++++++++++++++++++\n negotiator/default.h | 8 ++\n object.h | 3 +-\n 7 files changed, 292 insertions(+), 167 deletions(-)\n create mode 100644 fetch-negotiator.c\n create mode 100644 fetch-negotiator.h\n create mode 100644 negotiator/default.c\n create mode 100644 negotiator/default.h\n\ndiff --git a/Makefile b/Makefile\nindex 1d27f3636..96f84d1dc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -859,6 +859,7 @@ LIB_OBJS += ewah/ewah_bitmap.o\n LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec-cmd.o\n+LIB_OBJS += fetch-negotiator.o\n LIB_OBJS += fetch-object.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n@@ -891,6 +892,7 @@ LIB_OBJS += merge-blobs.o\n LIB_OBJS += merge-recursive.o\n LIB_OBJS += mergesort.o\n LIB_OBJS += name-hash.o\n+LIB_OBJS += negotiator/default.o\n LIB_OBJS += notes.o\n LIB_OBJS += notes-cache.o\n LIB_OBJS += notes-merge.o\ndiff --git a/fetch-negotiator.c b/fetch-negotiator.c\nnew file mode 100644\nindex 000000000..2675d120f\n--- /dev/null\n+++ b/fetch-negotiator.c\n@@ -0,0 +1,8 @@\n+#include \"git-compat-util.h\"\n+#include \"fetch-negotiator.h\"\n+#include \"negotiator/default.h\"\n+\n+void fetch_negotiator_init(struct fetch_negotiator *negotiator)\n+{\n+\tdefault_negotiator_init(negotiator);\n+}\ndiff --git a/fetch-negotiator.h b/fetch-negotiator.h\nnew file mode 100644\nindex 000000000..b1290aa9c\n--- /dev/null\n+++ b/fetch-negotiator.h\n@@ -0,0 +1,57 @@\n+#ifndef FETCH_NEGOTIATOR\n+#define FETCH_NEGOTIATOR\n+\n+struct commit;\n+\n+/*\n+ * An object that supplies the information needed to negotiate the contents of\n+ * the to-be-sent packfile during a fetch.\n+ *\n+ * To set up the negotiator, call fetch_negotiator_init(), then known_common()\n+ * (0 or more times), then add_tip() (0 or more times).\n+ *\n+ * Then, when \"have\" lines are required, call next(). Call ack() to report what\n+ * the server tells us.\n+ *\n+ * Once negotiation is done, call release(). The negotiator then cannot be used\n+ * (unless reinitialized with fetch_negotiator_init()).\n+ */\n+struct fetch_negotiator {\n+\t/*\n+\t * Before negotiation starts, indicate that the server is known to have\n+\t * this commit.\n+\t */\n+\tvoid (*known_common)(struct fetch_negotiator *, struct commit *);\n+\n+\t/*\n+\t * Once this function is invoked, known_common() cannot be invoked any\n+\t * more.\n+\t *\n+\t * Indicate that this commit and all its ancestors are to be checked\n+\t * for commonality with the server.\n+\t */\n+\tvoid (*add_tip)(struct fetch_negotiator *, struct commit *);\n+\n+\t/*\n+\t * Once this function is invoked, known_common() and add_tip() cannot\n+\t * be invoked any more.\n+\t *\n+\t * Return the next commit that the client should send as a \"have\" line.\n+\t */\n+\tconst struct object_id *(*next)(struct fetch_negotiator *);\n+\n+\t/*\n+\t * Inform the negotiator that the server has the given commit. This\n+\t * method must only be called on commits returned by next().\n+\t */\n+\tint (*ack)(struct fetch_negotiator *, struct commit *);\n+\n+\tvoid (*release)(struct fetch_negotiator *);\n+\n+\t/* internal use */\n+\tvoid *data;\n+};\n+\n+void fetch_negotiator_init(struct fetch_negotiator *negotiator);\n+\n+#endif\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex fb76d4017..4b13d3389 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -15,10 +15,10 @@\n #include \"connect.h\"\n #include \"transport.h\"\n #include \"version.h\"\n-#include \"prio-queue.h\"\n #include \"sha1-array.h\"\n #include \"oidset.h\"\n #include \"packfile.h\"\n+#include \"fetch-negotiator.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -36,13 +36,7 @@ static const char *alternate_shallow_file;\n \n /* Remember to update object flag allocation in object.h */\n #define COMPLETE\t(1U << 0)\n-#define COMMON\t\t(1U << 1)\n-#define COMMON_REF\t(1U << 2)\n-#define SEEN\t\t(1U << 3)\n-#define POPPED\t\t(1U << 4)\n-#define ALTERNATE\t(1U << 5)\n-\n-static int marked;\n+#define ALTERNATE\t(1U << 1)\n \n /*\n * After sending this many \"have\"s if we do not get any new ACK , we\n@@ -50,11 +44,6 @@ static int marked;\n */\n #define MAX_IN_VAIN 256\n \n-struct negotiation_state {\n-\tstruct prio_queue rev_list;\n-\tint non_common_revs;\n-};\n-\n static int multi_ack, use_sideband;\n /* Allow specifying sha1 if it is a ref tip. */\n #define ALLOW_TIP_SHA1\t01\n@@ -97,8 +86,8 @@ static void cache_one_alternate(const char *refname,\n \tcache->items[cache->nr++] = obj;\n }\n \n-static void for_each_cached_alternate(struct negotiation_state *ns,\n-\t\t\t\t void (*cb)(struct negotiation_state *,\n+static void for_each_cached_alternate(struct fetch_negotiator *negotiator,\n+\t\t\t\t void (*cb)(struct fetch_negotiator *,\n \t\t\t\t\t struct object *))\n {\n \tstatic int initialized;\n@@ -111,33 +100,17 @@ static void for_each_cached_alternate(struct negotiation_state *ns,\n \t}\n \n \tfor (i = 0; i < cache.nr; i++)\n-\t\tcb(ns, cache.items[i]);\n-}\n-\n-static void rev_list_push(struct negotiation_state *ns,\n-\t\t\t struct commit *commit, int mark)\n-{\n-\tif (!(commit->object.flags & mark)) {\n-\t\tcommit->object.flags |= mark;\n-\n-\t\tif (parse_commit(commit))\n-\t\t\treturn;\n-\n-\t\tprio_queue_put(&ns->rev_list, commit);\n-\n-\t\tif (!(commit->object.flags & COMMON))\n-\t\t\tns->non_common_revs++;\n-\t}\n+\t\tcb(negotiator, cache.items[i]);\n }\n \n-static int rev_list_insert_ref(struct negotiation_state *ns,\n+static int rev_list_insert_ref(struct fetch_negotiator *negotiator,\n \t\t\t const char *refname,\n \t\t\t const struct object_id *oid)\n {\n \tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n \n \tif (o && o->type == OBJ_COMMIT)\n-\t\trev_list_push(ns, (struct commit *)o, SEEN);\n+\t\tnegotiator->add_tip(negotiator, (struct commit *)o);\n \n \treturn 0;\n }\n@@ -148,98 +121,6 @@ static int rev_list_insert_ref_oid(const char *refname, const struct object_id *\n \treturn rev_list_insert_ref(cb_data, refname, oid);\n }\n \n-static int clear_marks(const char *refname, const struct object_id *oid,\n-\t\t int flag, void *cb_data)\n-{\n-\tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n-\n-\tif (o && o->type == OBJ_COMMIT)\n-\t\tclear_commit_marks((struct commit *)o,\n-\t\t\t\t COMMON | COMMON_REF | SEEN | POPPED);\n-\treturn 0;\n-}\n-\n-/*\n- This function marks a rev and its ancestors as common.\n- In some cases, it is desirable to mark only the ancestors (for example\n- when only the server does not yet know that they are common).\n-*/\n-\n-static void mark_common(struct negotiation_state *ns, struct commit *commit,\n-\t\tint ancestors_only, int dont_parse)\n-{\n-\tif (commit != NULL && !(commit->object.flags & COMMON)) {\n-\t\tstruct object *o = (struct object *)commit;\n-\n-\t\tif (!ancestors_only)\n-\t\t\to->flags |= COMMON;\n-\n-\t\tif (!(o->flags & SEEN))\n-\t\t\trev_list_push(ns, commit, SEEN);\n-\t\telse {\n-\t\t\tstruct commit_list *parents;\n-\n-\t\t\tif (!ancestors_only && !(o->flags & POPPED))\n-\t\t\t\tns->non_common_revs--;\n-\t\t\tif (!o->parsed && !dont_parse)\n-\t\t\t\tif (parse_commit(commit))\n-\t\t\t\t\treturn;\n-\n-\t\t\tfor (parents = commit->parents;\n-\t\t\t\t\tparents;\n-\t\t\t\t\tparents = parents->next)\n-\t\t\t\tmark_common(ns, parents->item, 0,\n-\t\t\t\t\t dont_parse);\n-\t\t}\n-\t}\n-}\n-\n-/*\n- Get the next rev to send, ignoring the common.\n-*/\n-\n-static const struct object_id *get_rev(struct negotiation_state *ns)\n-{\n-\tstruct commit *commit = NULL;\n-\n-\twhile (commit == NULL) {\n-\t\tunsigned int mark;\n-\t\tstruct commit_list *parents;\n-\n-\t\tif (ns->rev_list.nr == 0 || ns->non_common_revs == 0)\n-\t\t\treturn NULL;\n-\n-\t\tcommit = prio_queue_get(&ns->rev_list);\n-\t\tparse_commit(commit);\n-\t\tparents = commit->parents;\n-\n-\t\tcommit->object.flags |= POPPED;\n-\t\tif (!(commit->object.flags & COMMON))\n-\t\t\tns->non_common_revs--;\n-\n-\t\tif (commit->object.flags & COMMON) {\n-\t\t\t/* do not send \"have\", and ignore ancestors */\n-\t\t\tcommit = NULL;\n-\t\t\tmark = COMMON | SEEN;\n-\t\t} else if (commit->object.flags & COMMON_REF)\n-\t\t\t/* send \"have\", and ignore ancestors */\n-\t\t\tmark = COMMON | SEEN;\n-\t\telse\n-\t\t\t/* send \"have\", also for its ancestors */\n-\t\t\tmark = SEEN;\n-\n-\t\twhile (parents) {\n-\t\t\tif (!(parents->item->object.flags & SEEN))\n-\t\t\t\trev_list_push(ns, parents->item, mark);\n-\t\t\tif (mark & COMMON)\n-\t\t\t\tmark_common(ns, parents->item, 1, 0);\n-\t\t\tparents = parents->next;\n-\t\t}\n-\t}\n-\n-\treturn &commit->object.oid;\n-}\n-\n enum ack_type {\n \tNAK = 0,\n \tACK,\n@@ -306,10 +187,10 @@ static void send_request(struct fetch_pack_args *args,\n \t\twrite_or_die(fd, buf->buf, buf->len);\n }\n \n-static void insert_one_alternate_object(struct negotiation_state *ns,\n+static void insert_one_alternate_object(struct fetch_negotiator *negotiator,\n \t\t\t\t\tstruct object *obj)\n {\n-\trev_list_insert_ref(ns, NULL, &obj->oid);\n+\trev_list_insert_ref(negotiator, NULL, &obj->oid);\n }\n \n #define INITIAL_FLUSH 16\n@@ -332,7 +213,7 @@ static int next_flush(int stateless_rpc, int count)\n \treturn count;\n }\n \n-static int find_common(struct negotiation_state *ns,\n+static int find_common(struct fetch_negotiator *negotiator,\n \t\t struct fetch_pack_args *args,\n \t\t int fd[2], struct object_id *result_oid,\n \t\t struct ref *refs)\n@@ -349,8 +230,8 @@ static int find_common(struct negotiation_state *ns,\n \tif (args->stateless_rpc && multi_ack == 1)\n \t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n \n-\tfor_each_ref(rev_list_insert_ref_oid, ns);\n-\tfor_each_cached_alternate(ns, insert_one_alternate_object);\n+\tfor_each_ref(rev_list_insert_ref_oid, negotiator);\n+\tfor_each_cached_alternate(negotiator, insert_one_alternate_object);\n \n \tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\n@@ -468,7 +349,7 @@ static int find_common(struct negotiation_state *ns,\n \tretval = -1;\n \tif (args->no_dependents)\n \t\tgoto done;\n-\twhile ((oid = get_rev(ns))) {\n+\twhile ((oid = negotiator->next(negotiator))) {\n \t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tprint_verbose(args, \"have %s\", oid_to_hex(oid));\n \t\tin_vain++;\n@@ -508,8 +389,7 @@ static int find_common(struct negotiation_state *ns,\n \t\t\t\t\tint was_common;\n \t\t\t\t\tif (!commit)\n \t\t\t\t\t\tdie(_(\"invalid commit %s\"), oid_to_hex(result_oid));\n-\t\t\t\t\twas_common = commit->object.flags & COMMON;\n-\t\t\t\t\tmark_common(ns, commit, 0, 1);\n+\t\t\t\t\twas_common = negotiator->ack(negotiator, commit);\n \t\t\t\t\tif (args->stateless_rpc\n \t\t\t\t\t && ack == ACK_common\n \t\t\t\t\t && !was_common) {\n@@ -718,7 +598,7 @@ static void filter_refs(struct fetch_pack_args *args,\n \t*refs = newlist;\n }\n \n-static void mark_alternate_complete(struct negotiation_state *unused,\n+static void mark_alternate_complete(struct fetch_negotiator *unused,\n \t\t\t\t struct object *obj)\n {\n \tmark_complete(&obj->oid);\n@@ -756,7 +636,7 @@ static int add_loose_objects_to_set(const struct object_id *oid,\n * earliest commit time of the objects in refs that are commits and that we know\n * the commit time of.\n */\n-static void mark_complete_and_common_ref(struct negotiation_state *ns,\n+static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n \t\t\t\t\t struct fetch_pack_args *args,\n \t\t\t\t\t struct ref **refs)\n {\n@@ -825,12 +705,8 @@ static void mark_complete_and_common_ref(struct negotiation_state *ns,\n \t\t\tif (!o || o->type != OBJ_COMMIT || !(o->flags & COMPLETE))\n \t\t\t\tcontinue;\n \n-\t\t\tif (!(o->flags & SEEN)) {\n-\t\t\t\trev_list_push(ns, (struct commit *)o,\n-\t\t\t\t\t COMMON_REF | SEEN);\n-\n-\t\t\t\tmark_common(ns, (struct commit *)o, 1, 1);\n-\t\t\t}\n+\t\t\tnegotiator->known_common(negotiator,\n+\t\t\t\t\t\t (struct commit *)o);\n \t\t}\n \t}\n \n@@ -1012,7 +888,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tstruct object_id oid;\n \tconst char *agent_feature;\n \tint agent_len;\n-\tstruct negotiation_state ns = { { compare_commits_by_commit_date } };\n+\tstruct fetch_negotiator negotiator;\n+\tfetch_negotiator_init(&negotiator);\n \n \tsort_ref_list(&ref, ref_compare_name);\n \tQSORT(sought, nr_sought, cmp_ref_by_name);\n@@ -1085,16 +962,13 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n \n-\tif (marked)\n-\t\tfor_each_ref(clear_marks, NULL);\n-\tmarked = 1;\n-\tmark_complete_and_common_ref(&ns, args, &ref);\n+\tmark_complete_and_common_ref(&negotiator, args, &ref);\n \tfilter_refs(args, &ref, sought, nr_sought);\n \tif (everything_local(args, &ref)) {\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n-\tif (find_common(&ns, args, fd, &oid, ref) < 0)\n+\tif (find_common(&negotiator, args, fd, &oid, ref) < 0)\n \t\tif (!args->keep_pack)\n \t\t\t/* When cloning, it is not unusual to have\n \t\t\t * no common commit.\n@@ -1114,7 +988,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\tdie(_(\"git fetch-pack: fetch failed.\"));\n \n all_done:\n-\tclear_prio_queue(&ns.rev_list);\n+\tnegotiator.release(&negotiator);\n \treturn ref;\n }\n \n@@ -1176,14 +1050,15 @@ static void add_common(struct strbuf *req_buf, struct oidset *common)\n \t}\n }\n \n-static int add_haves(struct negotiation_state *ns, struct strbuf *req_buf,\n+static int add_haves(struct fetch_negotiator *negotiator,\n+\t\t struct strbuf *req_buf,\n \t\t int *haves_to_send, int *in_vain)\n {\n \tint ret = 0;\n \tint haves_added = 0;\n \tconst struct object_id *oid;\n \n-\twhile ((oid = get_rev(ns))) {\n+\twhile ((oid = negotiator->next(negotiator))) {\n \t\tpacket_buf_write(req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tif (++haves_added >= *haves_to_send)\n \t\t\tbreak;\n@@ -1202,7 +1077,7 @@ static int add_haves(struct negotiation_state *ns, struct strbuf *req_buf,\n \treturn ret;\n }\n \n-static int send_fetch_request(struct negotiation_state *ns, int fd_out,\n+static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,\n \t\t\t const struct fetch_pack_args *args,\n \t\t\t const struct ref *wants, struct oidset *common,\n \t\t\t int *haves_to_send, int *in_vain)\n@@ -1259,7 +1134,7 @@ static int send_fetch_request(struct negotiation_state *ns, int fd_out,\n \t\tadd_common(&req_buf, common);\n \n \t\t/* Add initial haves */\n-\t\tret = add_haves(ns, &req_buf, haves_to_send, in_vain);\n+\t\tret = add_haves(negotiator, &req_buf, haves_to_send, in_vain);\n \t}\n \n \t/* Send request */\n@@ -1296,7 +1171,7 @@ static int process_section_header(struct packet_reader *reader,\n \treturn ret;\n }\n \n-static int process_acks(struct negotiation_state *ns,\n+static int process_acks(struct fetch_negotiator *negotiator,\n \t\t\tstruct packet_reader *reader,\n \t\t\tstruct oidset *common)\n {\n@@ -1317,7 +1192,7 @@ static int process_acks(struct negotiation_state *ns,\n \t\t\t\tstruct commit *commit;\n \t\t\t\toidset_insert(common, &oid);\n \t\t\t\tcommit = lookup_commit(&oid);\n-\t\t\t\tmark_common(ns, commit, 0, 1);\n+\t\t\t\tnegotiator->ack(negotiator, commit);\n \t\t\t}\n \t\t\tcontinue;\n \t\t}\n@@ -1395,7 +1270,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \tstruct packet_reader reader;\n \tint in_vain = 0;\n \tint haves_to_send = INITIAL_FLUSH;\n-\tstruct negotiation_state ns = { { compare_commits_by_commit_date } };\n+\tstruct fetch_negotiator negotiator;\n+\tfetch_negotiator_init(&negotiator);\n \tpacket_reader_init(&reader, fd[0], NULL, 0,\n \t\t\t PACKET_READ_CHOMP_NEWLINE);\n \n@@ -1411,24 +1287,21 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n \t\t\t\targs->deepen = 1;\n \n-\t\t\tif (marked)\n-\t\t\t\tfor_each_ref(clear_marks, NULL);\n-\t\t\tmarked = 1;\n-\n \t\t\t/* Filter 'ref' by 'sought' and those that aren't local */\n-\t\t\tmark_complete_and_common_ref(&ns, args, &ref);\n+\t\t\tmark_complete_and_common_ref(&negotiator, args, &ref);\n \t\t\tfilter_refs(args, &ref, sought, nr_sought);\n \t\t\tif (everything_local(args, &ref))\n \t\t\t\tstate = FETCH_DONE;\n \t\t\telse\n \t\t\t\tstate = FETCH_SEND_REQUEST;\n \n-\t\t\tfor_each_ref(rev_list_insert_ref_oid, &ns);\n-\t\t\tfor_each_cached_alternate(&ns,\n+\t\t\tfor_each_ref(rev_list_insert_ref_oid, &negotiator);\n+\t\t\tfor_each_cached_alternate(&negotiator,\n \t\t\t\t\t\t insert_one_alternate_object);\n \t\t\tbreak;\n \t\tcase FETCH_SEND_REQUEST:\n-\t\t\tif (send_fetch_request(&ns, fd[1], args, ref, &common,\n+\t\t\tif (send_fetch_request(&negotiator, fd[1], args, ref,\n+\t\t\t\t\t &common,\n \t\t\t\t\t &haves_to_send, &in_vain))\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\telse\n@@ -1436,7 +1309,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t\tbreak;\n \t\tcase FETCH_PROCESS_ACKS:\n \t\t\t/* Process ACKs/NAKs */\n-\t\t\tswitch (process_acks(&ns, &reader, &common)) {\n+\t\t\tswitch (process_acks(&negotiator, &reader, &common)) {\n \t\t\tcase 2:\n \t\t\t\tstate = FETCH_GET_PACK;\n \t\t\t\tbreak;\n@@ -1465,7 +1338,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n \t\t}\n \t}\n \n-\tclear_prio_queue(&ns.rev_list);\n+\tnegotiator.release(&negotiator);\n \toidset_clear(&common);\n \treturn ref;\n }\ndiff --git a/negotiator/default.c b/negotiator/default.c\nnew file mode 100644\nindex 000000000..382fc7772\n--- /dev/null\n+++ b/negotiator/default.c\n@@ -0,0 +1,176 @@\n+#include \"cache.h\"\n+#include \"default.h\"\n+#include \"../commit.h\"\n+#include \"../fetch-negotiator.h\"\n+#include \"../prio-queue.h\"\n+#include \"../refs.h\"\n+#include \"../tag.h\"\n+\n+/* Remember to update object flag allocation in object.h */\n+#define COMMON\t\t(1U << 2)\n+#define COMMON_REF\t(1U << 3)\n+#define SEEN\t\t(1U << 4)\n+#define POPPED\t\t(1U << 5)\n+\n+static int marked;\n+\n+struct negotiation_state {\n+\tstruct prio_queue rev_list;\n+\tint non_common_revs;\n+};\n+\n+static void rev_list_push(struct negotiation_state *ns,\n+\t\t\t struct commit *commit, int mark)\n+{\n+\tif (!(commit->object.flags & mark)) {\n+\t\tcommit->object.flags |= mark;\n+\n+\t\tif (parse_commit(commit))\n+\t\t\treturn;\n+\n+\t\tprio_queue_put(&ns->rev_list, commit);\n+\n+\t\tif (!(commit->object.flags & COMMON))\n+\t\t\tns->non_common_revs++;\n+\t}\n+}\n+\n+static int clear_marks(const char *refname, const struct object_id *oid,\n+\t\t int flag, void *cb_data)\n+{\n+\tstruct object *o = deref_tag(parse_object(oid), refname, 0);\n+\n+\tif (o && o->type == OBJ_COMMIT)\n+\t\tclear_commit_marks((struct commit *)o,\n+\t\t\t\t COMMON | COMMON_REF | SEEN | POPPED);\n+\treturn 0;\n+}\n+\n+/*\n+ * This function marks a rev and its ancestors as common.\n+ * In some cases, it is desirable to mark only the ancestors (for example\n+ * when only the server does not yet know that they are common).\n+ */\n+static void mark_common(struct negotiation_state *ns, struct commit *commit,\n+\t\tint ancestors_only, int dont_parse)\n+{\n+\tif (commit != NULL && !(commit->object.flags & COMMON)) {\n+\t\tstruct object *o = (struct object *)commit;\n+\n+\t\tif (!ancestors_only)\n+\t\t\to->flags |= COMMON;\n+\n+\t\tif (!(o->flags & SEEN))\n+\t\t\trev_list_push(ns, commit, SEEN);\n+\t\telse {\n+\t\t\tstruct commit_list *parents;\n+\n+\t\t\tif (!ancestors_only && !(o->flags & POPPED))\n+\t\t\t\tns->non_common_revs--;\n+\t\t\tif (!o->parsed && !dont_parse)\n+\t\t\t\tif (parse_commit(commit))\n+\t\t\t\t\treturn;\n+\n+\t\t\tfor (parents = commit->parents;\n+\t\t\t\t\tparents;\n+\t\t\t\t\tparents = parents->next)\n+\t\t\t\tmark_common(ns, parents->item, 0,\n+\t\t\t\t\t dont_parse);\n+\t\t}\n+\t}\n+}\n+\n+/*\n+ * Get the next rev to send, ignoring the common.\n+ */\n+static const struct object_id *get_rev(struct negotiation_state *ns)\n+{\n+\tstruct commit *commit = NULL;\n+\n+\twhile (commit == NULL) {\n+\t\tunsigned int mark;\n+\t\tstruct commit_list *parents;\n+\n+\t\tif (ns->rev_list.nr == 0 || ns->non_common_revs == 0)\n+\t\t\treturn NULL;\n+\n+\t\tcommit = prio_queue_get(&ns->rev_list);\n+\t\tparse_commit(commit);\n+\t\tparents = commit->parents;\n+\n+\t\tcommit->object.flags |= POPPED;\n+\t\tif (!(commit->object.flags & COMMON))\n+\t\t\tns->non_common_revs--;\n+\n+\t\tif (commit->object.flags & COMMON) {\n+\t\t\t/* do not send \"have\", and ignore ancestors */\n+\t\t\tcommit = NULL;\n+\t\t\tmark = COMMON | SEEN;\n+\t\t} else if (commit->object.flags & COMMON_REF)\n+\t\t\t/* send \"have\", and ignore ancestors */\n+\t\t\tmark = COMMON | SEEN;\n+\t\telse\n+\t\t\t/* send \"have\", also for its ancestors */\n+\t\t\tmark = SEEN;\n+\n+\t\twhile (parents) {\n+\t\t\tif (!(parents->item->object.flags & SEEN))\n+\t\t\t\trev_list_push(ns, parents->item, mark);\n+\t\t\tif (mark & COMMON)\n+\t\t\t\tmark_common(ns, parents->item, 1, 0);\n+\t\t\tparents = parents->next;\n+\t\t}\n+\t}\n+\n+\treturn &commit->object.oid;\n+}\n+\n+static void known_common(struct fetch_negotiator *n, struct commit *c)\n+{\n+\tif (!(c->object.flags & SEEN)) {\n+\t\trev_list_push(n->data, c, COMMON_REF | SEEN);\n+\t\tmark_common(n->data, c, 1, 1);\n+\t}\n+}\n+\n+static void add_tip(struct fetch_negotiator *n, struct commit *c)\n+{\n+\tn->known_common = NULL;\n+\trev_list_push(n->data, c, SEEN);\n+}\n+\n+static const struct object_id *next(struct fetch_negotiator *n)\n+{\n+\tn->known_common = NULL;\n+\tn->add_tip = NULL;\n+\treturn get_rev(n->data);\n+}\n+\n+static int ack(struct fetch_negotiator *n, struct commit *c)\n+{\n+\tint known_to_be_common = !!(c->object.flags & COMMON);\n+\tmark_common(n->data, c, 0, 1);\n+\treturn known_to_be_common;\n+}\n+\n+static void release(struct fetch_negotiator *n)\n+{\n+\tclear_prio_queue(&((struct negotiation_state *)n->data)->rev_list);\n+\tFREE_AND_NULL(n->data);\n+}\n+\n+void default_negotiator_init(struct fetch_negotiator *negotiator)\n+{\n+\tstruct negotiation_state *ns;\n+\tnegotiator->known_common = known_common;\n+\tnegotiator->add_tip = add_tip;\n+\tnegotiator->next = next;\n+\tnegotiator->ack = ack;\n+\tnegotiator->release = release;\n+\tnegotiator->data = ns = xcalloc(1, sizeof(*ns));\n+\tns->rev_list.compare = compare_commits_by_commit_date;\n+\n+\tif (marked)\n+\t\tfor_each_ref(clear_marks, NULL);\n+\tmarked = 1;\n+}\ndiff --git a/negotiator/default.h b/negotiator/default.h\nnew file mode 100644\nindex 000000000..d23a8f2fb\n--- /dev/null\n+++ b/negotiator/default.h\n@@ -0,0 +1,8 @@\n+#ifndef NEGOTIATOR_DEFAULT_H\n+#define NEGOTIATOR_DEFAULT_H\n+\n+struct fetch_negotiator;\n+\n+void default_negotiator_init(struct fetch_negotiator *negotiator);\n+\n+#endif\ndiff --git a/object.h b/object.h\nindex 5c1395500..7db4941d6 100644\n--- a/object.h\n+++ b/object.h\n@@ -28,7 +28,8 @@ struct object_array {\n /*\n * object flag allocation:\n * revision.h: 0---------10 26\n- * fetch-pack.c: 0----5\n+ * fetch-pack.c: 01\n+ * negotiator/default.c: 2--5\n * walker.c: 0-2\n * upload-pack.c: 4 11----------------19\n * builtin/blame.c: 12-13\n-- \n2.17.0.582.gccdcbd54c4\n\n"},{"id":"350265","messageId":"xmqqefh82g05.fsf@gitster-ct.c.googlers.com","threadId":"48648","inReplyTo":"fdcd88ba460d1feddb40037ff5b935339a51368b.1529016350.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 3/7] fetch-pack: directly end negotiation if ACK ready","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-15T16:04:10Z","receivedAt":"2018-06-15T16:04:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> When \"ACK %s ready\" is received, find_common() clears rev_list in an\n> attempt to stop further \"have\" lines from being sent [1]. It is much\n> more readable to explicitly break from the loop instead.\n>\n> So explicitly break from the loop, and make the clearing of the rev_list\n> happen unconditionally.\n>\n> [1] The rationale is further described in the originating commit\n> f2cba9299b (\"fetch-pack: Finish negotation if remote replies \"ACK %s\n> ready\"\", 2011-03-14).\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> fetch-pack.c | 9 +++++----\n> 1 file changed, 5 insertions(+), 4 deletions(-)\n\nThis makes more sense to me; I was wondering where the prio-queue\ngets cleared, or leaking it is OK, while reading the previous round.\n\nThanks.\n\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index 2812499a5..60adfc073 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -517,10 +517,8 @@ static int find_common(struct fetch_pack_args *args,\n> \t\t\t\t\tmark_common(commit, 0, 1);\n> \t\t\t\t\tretval = 0;\n> \t\t\t\t\tgot_continue = 1;\n> -\t\t\t\t\tif (ack == ACK_ready) {\n> -\t\t\t\t\t\tclear_prio_queue(&rev_list);\n> +\t\t\t\t\tif (ack == ACK_ready)\n> \t\t\t\t\t\tgot_ready = 1;\n> -\t\t\t\t\t}\n> \t\t\t\t\tbreak;\n> \t\t\t\t\t}\n> \t\t\t\t}\n> @@ -530,6 +528,8 @@ static int find_common(struct fetch_pack_args *args,\n> \t\t\t\tprint_verbose(args, _(\"giving up\"));\n> \t\t\t\tbreak; /* give up */\n> \t\t\t}\n> +\t\t\tif (got_ready)\n> +\t\t\t\tbreak;\n> \t\t}\n> \t}\n> done:\n> @@ -1096,6 +1096,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n> \t\tdie(_(\"git fetch-pack: fetch failed.\"));\n> \n> all_done:\n> +\tclear_prio_queue(&rev_list);\n> \treturn ref;\n> }\n> \n> @@ -1300,7 +1301,6 @@ static int process_acks(struct packet_reader *reader, struct oidset *common)\n> \t\t}\n> \n> \t\tif (!strcmp(reader->line, \"ready\")) {\n> -\t\t\tclear_prio_queue(&rev_list);\n> \t\t\treceived_ready = 1;\n> \t\t\tcontinue;\n> \t\t}\n> @@ -1441,6 +1441,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,\n> \t\t}\n> \t}\n> \n> +\tclear_prio_queue(&rev_list);\n> \toidset_clear(&common);\n> \treturn ref;\n> }\n"},{"id":"350878","messageId":"20180625182434.GE19910@google.com","threadId":"48648","inReplyTo":"cover.1529016350.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 0/7] Refactor fetch negotiation into its own API","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-06-25T18:24:34Z","receivedAt":"2018-06-25T18:24:40Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 06/14, Jonathan Tan wrote:\n> Thanks, Brandon and Junio, for your comments.\n> \n> Updates since v2:\n> - nothing new in patches 1 and 2\n> - patch 3: now clear priority queue unconditionally once we are done\n> with it (as requested by Brandon in a comment for a later patch)\n> - patch 4: updated commit message to not mention everything_local() (as\n> pointed out by Brandon), updated test to not rely on the fact that\n> fetch-pack uses prefix matching (thanks, Junio, for the observation)\n> - patch 5: used a more descriptive name (\"struct negotiation_state\")\n> for the struct, instead of \"struct data\"\n> - squashed patch 8 into patch 7; this means that the comments are not\n> moved verbatim, but for the reviewer, verbatim-ness of comments is\n> probably not that important anyway\n\nThanks this series looks good now.\n\nReviewed-by: Brandon Williams \n\n-- \nBrandon Williams\n"}]}