{"thread":{"id":"51705","subject":"[PATCH 0/2] Skip ls-refs if possible for HTTP","startedAt":"2019-08-21T22:20:15Z","lastAt":"2019-08-22T17:44:30Z","messageCount":5,"participants":["Jonathan Tan","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"380912","messageId":"cover.1566425828.git.jonathantanmy@google.com","threadId":"51705","inReplyTo":null,"subject":"[PATCH 0/2] Skip ls-refs if possible for HTTP","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-08-21T22:20:08Z","receivedAt":"2019-08-21T22:20:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This was discovered by one of my colleagues when using a partial clone.\nI thought I had resolved the problem with the commits mentioned in patch\n1 (e70a3030e7 and ancestors), but apparently that is not the case (that\nonly worked for native protocols). So here is a fix for HTTP.\n\nI'm not sure of the value of the test in patch 2, but that test does\nfail if I don't update fetch_refs_from_bundle() to first call\nget_refs_from_bundle() if it hasn't already been called.\n\nJonathan Tan (2):\n  transport-helper: skip ls-refs if unnecessary\n  transport: teach all vtables to allow fetch first\n\n t/t5607-clone-bundle.sh | 11 +++++++++++\n t/t5702-protocol-v2.sh  | 13 +++++++++++++\n transport-helper.c      | 38 ++++++++++++++++++++++++++++++++------\n transport-internal.h    |  6 ------\n transport.c             | 18 ++++++------------\n 5 files changed, 62 insertions(+), 24 deletions(-)\n\n-- \n2.23.0.187.g17f5b7556c-goog\n\n"},{"id":"380913","messageId":"9e0971d10117651a9a475c3b5c183a09e617a67b.1566425828.git.jonathantanmy@google.com","threadId":"51705","inReplyTo":"cover.1566425828.git.jonathantanmy@google.com","subject":"[PATCH 1/2] transport-helper: skip ls-refs if unnecessary","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-08-21T22:20:09Z","receivedAt":"2019-08-21T22:20:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Commit e70a3030e7 (\"fetch: do not list refs if fetching only hashes\",\n2018-10-07) and its ancestors taught Git, as an optimization, to skip\nthe ls-refs step when it is not necessary during a protocol v2 fetch\n(for example, when lazy fetching a missing object in a partial clone, or\nwhen running \"git fetch --no-tags <remote> <SHA-1>\"). But that was only\ndone for natively supported protocols; in particular, HTTP was not\nsupported.\n\nTeach Git to skip ls-refs when using remote helpers that support connect\nor stateless-connect. To do this, fetch() is made an acceptable entry\npoint. Because fetch() can now be the first function in the vtable\ncalled, \"get_helper(transport);\" has to be added to the beginning of\nthat function to set the transport up (if not yet set up) before\nprocess_connect() is invoked.\n\nWhen fetch() is called, the transport could be taken over (this happens\nif \"connect\" or \"stateless-connect\" is successfully run without any\n\"fallback\" response), or not. If the transport is taken over, execution\ncontinues like execution for natively supported protocols\n(fetch_refs_via_pack() is executed, which will fetch refs using ls-refs\nif needed). If not, the remote helper interface will invoke\nget_refs_list() if it hasn't been invoked yet, preserving existing\nbehavior.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n t/t5702-protocol-v2.sh | 13 +++++++++++++\n transport-helper.c     | 39 +++++++++++++++++++++++++++++++++------\n 2 files changed, 46 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh\nindex 011b81d4fc..61f5080de0 100755\n--- a/t/t5702-protocol-v2.sh\n+++ b/t/t5702-protocol-v2.sh\n@@ -631,6 +631,19 @@ test_expect_success 'fetch with http:// using protocol v2' '\n \tgrep \"git< version 2\" log\n '\n \n+test_expect_success 'fetch with http:// by hash without tag following with protocol v2 does not list refs' '\n+\ttest_when_finished \"rm -f log\" &&\n+\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" two_a &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" rev-parse two_a >two_a_hash &&\n+\n+\tGIT_TRACE_PACKET=\"$(pwd)/log\" git -C http_child -c protocol.version=2 \\\n+\t\tfetch --no-tags origin $(cat two_a_hash) &&\n+\n+\tgrep \"fetch< version 2\" log &&\n+\t! grep \"fetch> command=ls-refs\" log\n+'\n+\n test_expect_success 'fetch from namespaced repo respects namespaces' '\n \ttest_when_finished \"rm -f log\" &&\n \ndiff --git a/transport-helper.c b/transport-helper.c\nindex 6b05a88faf..1fb31e1a6e 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -33,6 +33,16 @@ struct helper_data {\n \t\tcheck_connectivity : 1,\n \t\tno_disconnect_req : 1,\n \t\tno_private_update : 1;\n+\n+\t/*\n+\t * As an optimization, the transport code may invoke fetch before\n+\t * get_refs_list. If this happens, and if the transport helper doesn't\n+\t * support connect or stateless_connect, we need to invoke\n+\t * get_refs_list ourselves if we haven't already done so. Keep track of\n+\t * whether we have invoked get_refs_list.\n+\t */\n+\tunsigned get_refs_list_called : 1;\n+\n \tchar *export_marks;\n \tchar *import_marks;\n \t/* These go from remote name (as in \"list\") to private name */\n@@ -652,17 +662,25 @@ static int connect_helper(struct transport *transport, const char *name,\n \treturn 0;\n }\n \n+static struct ref *get_refs_list_using_list(struct transport *transport,\n+\t\t\t\t\t    int for_push);\n+\n static int fetch(struct transport *transport,\n \t\t int nr_heads, struct ref **to_fetch)\n {\n \tstruct helper_data *data = transport->data;\n \tint i, count;\n \n+\tget_helper(transport);\n+\n \tif (process_connect(transport, 0)) {\n \t\tdo_take_over(transport);\n \t\treturn transport->vtable->fetch(transport, nr_heads, to_fetch);\n \t}\n \n+\tif (!data->get_refs_list_called)\n+\t\tget_refs_list_using_list(transport, 0);\n+\n \tcount = 0;\n \tfor (i = 0; i < nr_heads; i++)\n \t\tif (!(to_fetch[i]->status & REF_STATUS_UPTODATE))\n@@ -1058,6 +1076,19 @@ static int has_attribute(const char *attrs, const char *attr)\n \n static struct ref *get_refs_list(struct transport *transport, int for_push,\n \t\t\t\t const struct argv_array *ref_prefixes)\n+{\n+\tget_helper(transport);\n+\n+\tif (process_connect(transport, for_push)) {\n+\t\tdo_take_over(transport);\n+\t\treturn transport->vtable->get_refs_list(transport, for_push, ref_prefixes);\n+\t}\n+\n+\treturn get_refs_list_using_list(transport, for_push);\n+}\n+\n+static struct ref *get_refs_list_using_list(struct transport *transport,\n+\t\t\t\t\t    int for_push)\n {\n \tstruct helper_data *data = transport->data;\n \tstruct child_process *helper;\n@@ -1066,13 +1097,9 @@ static struct ref *get_refs_list(struct transport *transport, int for_push,\n \tstruct ref *posn;\n \tstruct strbuf buf = STRBUF_INIT;\n \n+\tdata->get_refs_list_called = 1;\n \thelper = get_helper(transport);\n \n-\tif (process_connect(transport, for_push)) {\n-\t\tdo_take_over(transport);\n-\t\treturn transport->vtable->get_refs_list(transport, for_push, ref_prefixes);\n-\t}\n-\n \tif (data->push && for_push)\n \t\twrite_str_in_full(helper->in, \"list for-push\\n\");\n \telse\n@@ -1119,7 +1146,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push,\n }\n \n static struct transport_vtable vtable = {\n-\t0,\n+\t1,\n \tset_helper_option,\n \tget_refs_list,\n \tfetch,\n-- \n2.23.0.187.g17f5b7556c-goog\n\n"},{"id":"380914","messageId":"5795a2a156af01966168176f9540ed1a8a7bb222.1566425828.git.jonathantanmy@google.com","threadId":"51705","inReplyTo":"cover.1566425828.git.jonathantanmy@google.com","subject":"[PATCH 2/2] transport: teach all vtables to allow fetch first","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-08-21T22:20:10Z","receivedAt":"2019-08-21T22:20:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The only transport that does not allow fetch() to be called before\nget_refs_list() is the bundle transport. Clean up the code by teaching\nthe bundle transport the ability to do this, and removing support for\ntransports that don't support this order of invocation.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n t/t5607-clone-bundle.sh | 11 +++++++++++\n transport-helper.c      |  1 -\n transport-internal.h    |  6 ------\n transport.c             | 18 ++++++------------\n 4 files changed, 17 insertions(+), 19 deletions(-)\n\ndiff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh\nindex 2a0fb15cf1..b7a3fdf02d 100755\n--- a/t/t5607-clone-bundle.sh\n+++ b/t/t5607-clone-bundle.sh\n@@ -83,4 +83,15 @@ test_expect_success 'failed bundle creation does not leave cruft' '\n \ttest_path_is_missing fail.bundle.lock\n '\n \n+test_expect_success 'fetch SHA-1 from bundle' '\n+\ttest_create_repo foo &&\n+\ttest_commit -C foo x &&\n+\tgit -C foo bundle create tip.bundle -1 master &&\n+\tgit -C foo rev-parse HEAD >hash &&\n+\n+\t# Exercise to ensure that fetching a SHA-1 from a bundle works with no\n+\t# errors\n+\tgit fetch --no-tags foo/tip.bundle \"$(cat hash)\"\n+'\n+\n test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 1fb31e1a6e..96955d4004 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -1146,7 +1146,6 @@ static struct ref *get_refs_list_using_list(struct transport *transport,\n }\n \n static struct transport_vtable vtable = {\n-\t1,\n \tset_helper_option,\n \tget_refs_list,\n \tfetch,\ndiff --git a/transport-internal.h b/transport-internal.h\nindex 004bee5e36..1cde6258a7 100644\n--- a/transport-internal.h\n+++ b/transport-internal.h\n@@ -6,12 +6,6 @@ struct transport;\n struct argv_array;\n \n struct transport_vtable {\n-\t/**\n-\t * This transport supports the fetch() function being called\n-\t * without get_refs_list() first being called.\n-\t */\n-\tunsigned fetch_without_list : 1;\n-\n \t/**\n \t * Returns 0 if successful, positive if the option is not\n \t * recognized or is inapplicable, and negative if the option\ndiff --git a/transport.c b/transport.c\nindex 778c60bf57..662a2d9ae0 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -122,6 +122,7 @@ static void set_upstreams(struct transport *transport, struct ref *refs,\n struct bundle_transport_data {\n \tint fd;\n \tstruct bundle_header header;\n+\tunsigned get_refs_from_bundle_called : 1;\n };\n \n static struct ref *get_refs_from_bundle(struct transport *transport,\n@@ -135,6 +136,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport,\n \tif (for_push)\n \t\treturn NULL;\n \n+\tdata->get_refs_from_bundle_called = 1;\n+\n \tif (data->fd > 0)\n \t\tclose(data->fd);\n \tdata->fd = read_bundle_header(transport->url, &data->header);\n@@ -154,6 +157,9 @@ static int fetch_refs_from_bundle(struct transport *transport,\n \t\t\t       int nr_heads, struct ref **to_fetch)\n {\n \tstruct bundle_transport_data *data = transport->data;\n+\n+\tif (!data->get_refs_from_bundle_called)\n+\t\tget_refs_from_bundle(transport, 0, NULL);\n \treturn unbundle(the_repository, &data->header, data->fd,\n \t\t\ttransport->progress ? BUNDLE_VERBOSE : 0);\n }\n@@ -742,7 +748,6 @@ static int disconnect_git(struct transport *transport)\n }\n \n static struct transport_vtable taken_over_vtable = {\n-\t1,\n \tNULL,\n \tget_refs_via_connect,\n \tfetch_refs_via_pack,\n@@ -892,7 +897,6 @@ void transport_check_allowed(const char *type)\n }\n \n static struct transport_vtable bundle_vtable = {\n-\t0,\n \tNULL,\n \tget_refs_from_bundle,\n \tfetch_refs_from_bundle,\n@@ -902,7 +906,6 @@ static struct transport_vtable bundle_vtable = {\n };\n \n static struct transport_vtable builtin_smart_vtable = {\n-\t1,\n \tNULL,\n \tget_refs_via_connect,\n \tfetch_refs_via_pack,\n@@ -1285,15 +1288,6 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)\n \tstruct ref **heads = NULL;\n \tstruct ref *rm;\n \n-\tif (!transport->vtable->fetch_without_list)\n-\t\t/*\n-\t\t * Some transports (e.g. the built-in bundle transport and the\n-\t\t * transport helper interface) do not work when fetching is\n-\t\t * done immediately after transport creation. List the remote\n-\t\t * refs anyway (if not already listed) as a workaround.\n-\t\t */\n-\t\ttransport_get_remote_refs(transport, NULL);\n-\n \tfor (rm = refs; rm; rm = rm->next) {\n \t\tnr_refs++;\n \t\tif (rm->peer_ref &&\n-- \n2.23.0.187.g17f5b7556c-goog\n\n"},{"id":"380948","messageId":"xmqqtva9p3ma.fsf@gitster-ct.c.googlers.com","threadId":"51705","inReplyTo":"cover.1566425828.git.jonathantanmy@google.com","subject":"Re: [PATCH 0/2] Skip ls-refs if possible for HTTP","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-08-22T16:27:25Z","receivedAt":"2019-08-22T16:27:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> This was discovered by one of my colleagues when using a partial clone.\n> I thought I had resolved the problem with the commits mentioned in patch\n> 1 (e70a3030e7 and ancestors), but apparently that is not the case (that\n> only worked for native protocols). So here is a fix for HTTP.\n>\n> I'm not sure of the value of the test in patch 2, but that test does\n> fail if I don't update fetch_refs_from_bundle() to first call\n> get_refs_from_bundle() if it hasn't already been called.\n\nThis probably is totally off-tangent, but do any of these \"let's\nadvertise fewer\" changes at the protocol level have to take into\naccount the use of --prune option on the client side?\n\n> Jonathan Tan (2):\n>   transport-helper: skip ls-refs if unnecessary\n>   transport: teach all vtables to allow fetch first\n>\n>  t/t5607-clone-bundle.sh | 11 +++++++++++\n>  t/t5702-protocol-v2.sh  | 13 +++++++++++++\n>  transport-helper.c      | 38 ++++++++++++++++++++++++++++++++------\n>  transport-internal.h    |  6 ------\n>  transport.c             | 18 ++++++------------\n>  5 files changed, 62 insertions(+), 24 deletions(-)\n"},{"id":"380959","messageId":"20190822172312.41404-1-jonathantanmy@google.com","threadId":"51705","inReplyTo":"xmqqtva9p3ma.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 0/2] Skip ls-refs if possible for HTTP","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-08-22T17:23:12Z","receivedAt":"2019-08-22T17:44:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> This probably is totally off-tangent, but do any of these \"let's\n> advertise fewer\" changes at the protocol level have to take into\n> account the use of --prune option on the client side?\n\nI don't think so. According to what I understand from the documentation,\nthe prune option prunes based on the RHS of the refspec, and it doesn't\naffect anything non-matching. When we advertise fewer, the only refs\nthat are missing are those that are non-matching anyway.\n\nSome experimentation seems to show that that is the case also:\n\n$ git init one\n[snip]\n$ git -C one commit --allow-empty -m x\n[master (root-commit) 67056ac] x\n$ git -C one branch maste\n$ git -C one branch other\n$ git clone \"$(pwd)/one\" two\nCloning into 'two'...\ndone.\n$ git -C one branch -d maste\nDeleted branch maste (was 67056ac).\n$ git -C two checkout --detach HEAD\nHEAD is now at 67056ac x\n$ GIT_TRACE_PACKET=1 git -c protocol.version=2 -C two fetch --prune origin refs/heads/m*:refs/remotes/origin/m*\n[snipped lots of stuff; basically, only refs/heads/master is sent]\n10:15:46.308264 pkt-line.c:80           packet:        fetch> ref-prefix refs/heads/m\n10:15:46.308276 pkt-line.c:80           packet:        fetch> ref-prefix refs/tags/\n10:15:46.308523 pkt-line.c:80           packet:        fetch< 67056ac6d07814334716df760054ac5bec05b66a refs/heads/master\nFrom /usr/local/google/home/jonathantanmy/tmp/g/one\n - [deleted]         (none)     -> origin/maste\n$ git -C two for-each-ref\n67056ac6d07814334716df760054ac5bec05b66a commit\trefs/heads/master\n67056ac6d07814334716df760054ac5bec05b66a commit\trefs/remotes/origin/HEAD\n67056ac6d07814334716df760054ac5bec05b66a commit\trefs/remotes/origin/master\n67056ac6d07814334716df760054ac5bec05b66a commit\trefs/remotes/origin/other\n\n(Notice that refs/remotes/origin/other is untouched.)\n"}]}