threads / patch / 42183

patch, 14 partsProtocol v2 patches

Subject: [WIP PATCH 00/14] Protocol v2 patches

## tl;dr

40 messages between Apr 29, 2016 and May 25, 2016. Diffs are folded; open one to read it.

replies: 39people: 5as markdown or json

Stefan Beller· Apr 29, 2016, 23:34 UTC · lore
Hi David,
here are my patches for a protocol v2.
("Negotiate capabilities before doing anything else", or as code:
        static void upload_pack_version_2(void)
        {
                send_capabilities_version_2();
                receive_capabilities_version_2();
                /* The rest of the protocol stays the same, capabilities advertising
                   is disabled though. */
                advertise_capabilities = 0;
                upload_pack();
        }
)

They are rough and unfinished as you can see by the tailing WIPs. However the plumbing (upload-pack and fetch-pack) works and we'd need to integrate that into user porcelains, i.e. fetch, clone, push.

Also we need to add tests for all the options again, so we'd need to be smart about testing that.

I am not sure if it makes sense to integrate that with the http series, though.

Thanks, Stefan

Nguyễn Thái Ngọc Duy (1):
  upload-pack: make client capability parsing code a separate function
Stefan Beller (13):
  upload-pack.c: Refactor capability advertising
  upload-pack-2: Implement the version 2 of upload-pack
  connect: rewrite feature parsing to work on string_list
  transport: add infrastructure to support a protocol version number
  remote.h: add get_remote_capabilities, request_capabilities
  fetch-pack: move capability selection out of do_fetch_pack
  fetch-pack: factor out get_selected_capabilities_list
  fetch-pack: Add negotiate_capabilities
  do_fetch_pack: select capabilities for transport version 1 only
  builtin/fetch-pack: add argument for transport version
  Add test for fetch-pack
  WIP add test for git pull
  WIP test git fetch
 .gitignore             |   1 +
 Makefile               |   4 ++
 builtin/fetch-pack.c   |  20 ++++++-
 builtin/receive-pack.c |  15 +++--
 connect.c              | 141 +++++++++++++++++++++++++++++--------------
 connect.h              |   2 +-
 fetch-pack.c           | 102 ++++++++++++++++++++++++-------
 fetch-pack.h           |   7 +++
 remote.c               |   2 +
 remote.h               |   5 ++
 t/t5500-fetch-pack.sh  |  21 +++++++
 t/t5510-fetch.sh       |   5 ++
 t/t5520-pull.sh        |   6 ++
 transport-helper.c     |   1 +
 transport.c            |  20 ++++++-
 transport.h            |   8 +++
 upload-pack-2.c        |   1 +
 upload-pack.c          | 159 ++++++++++++++++++++++++++++++++++++-------------
 18 files changed, 403 insertions(+), 117 deletions(-)
 create mode 120000 upload-pack-2.c
-- 
2.8.0.32.g71f8beb.dirty
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 01/14] upload-pack: make client capability parsing code a separate function

From: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 upload-pack.c | 44 +++++++++++++++++++++++---------------------
 1 file changed, 23 insertions(+), 21 deletions(-)
Show changes to upload-pack.c +23 −21
diff --git a/upload-pack.c b/upload-pack.c
index dc802a0..aaaf883 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -535,6 +535,28 @@ error:
 	}
 }
 
+static void parse_features(const char *features)
+{
+	if (parse_feature_request(features, "multi_ack_detailed"))
+		multi_ack = 2;
+	else if (parse_feature_request(features, "multi_ack"))
+		multi_ack = 1;
+	if (parse_feature_request(features, "no-done"))
+		no_done = 1;
+	if (parse_feature_request(features, "thin-pack"))
+		use_thin_pack = 1;
+	if (parse_feature_request(features, "ofs-delta"))
+		use_ofs_delta = 1;
+	if (parse_feature_request(features, "side-band-64k"))
+		use_sideband = LARGE_PACKET_MAX;
+	else if (parse_feature_request(features, "side-band"))
+		use_sideband = DEFAULT_PACKET_MAX;
+	if (parse_feature_request(features, "no-progress"))
+		no_progress = 1;
+	if (parse_feature_request(features, "include-tag"))
+		use_include_tag = 1;
+}
+
 static void receive_needs(void)
 {
 	struct object_array shallows = OBJECT_ARRAY_INIT;
@@ -544,7 +566,6 @@ static void receive_needs(void)
 	shallow_nr = 0;
 	for (;;) {
 		struct object *o;
-		const char *features;
 		unsigned char sha1_buf[20];
 		char *line = packet_read_line(0, NULL);
 		reset_timeout();
@@ -579,26 +600,7 @@ static void receive_needs(void)
 			die("git upload-pack: protocol error, "
 			    "expected to get sha, not '%s'", line);
 
-		features = line + 45;
-
-		if (parse_feature_request(features, "multi_ack_detailed"))
-			multi_ack = 2;
-		else if (parse_feature_request(features, "multi_ack"))
-			multi_ack = 1;
-		if (parse_feature_request(features, "no-done"))
-			no_done = 1;
-		if (parse_feature_request(features, "thin-pack"))
-			use_thin_pack = 1;
-		if (parse_feature_request(features, "ofs-delta"))
-			use_ofs_delta = 1;
-		if (parse_feature_request(features, "side-band-64k"))
-			use_sideband = LARGE_PACKET_MAX;
-		else if (parse_feature_request(features, "side-band"))
-			use_sideband = DEFAULT_PACKET_MAX;
-		if (parse_feature_request(features, "no-progress"))
-			no_progress = 1;
-		if (parse_feature_request(features, "include-tag"))
-			use_include_tag = 1;
+		parse_features(line + 45);
 
 		o = parse_object(sha1_buf);
 		if (!o)
-- 
2.8.0.32.g71f8beb.dirty
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 02/14] upload-pack.c: Refactor capability advertising

Instead of having the capabilities in a local string, keep them in a struct outside the function. This will allow us in a later patch to easily reuse the capabilities in version 2 of the protocol.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 upload-pack.c | 59 ++++++++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 40 insertions(+), 19 deletions(-)
Show changes to upload-pack.c +40 −19
diff --git a/upload-pack.c b/upload-pack.c
index aaaf883..85381d5 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -719,37 +719,58 @@ static void format_symref_info(struct strbuf *buf, struct string_list *symref)
 		strbuf_addf(buf, " symref=%s:%s", item->string, (char *)item->util);
 }
 
+static int advertise_capabilities = 1;
+const char *all_capabilities[] = {
+	"multi_ack",
+	"thin-pack",
+	"side-band",
+	"side-band-64k",
+	"ofs-delta",
+	"shallow",
+	"no-progress",
+	"include-tag",
+	"multi_ack_detailed",
+	"allow-tip-sha1-in-want",
+	"allow-reachable-sha1-in-want",
+	"no-done",
+};
+
 static int send_ref(const char *refname, const struct object_id *oid,
 		    int flag, void *cb_data)
 {
-	static const char *capabilities = "multi_ack thin-pack side-band"
-		" side-band-64k ofs-delta shallow no-progress"
-		" include-tag multi_ack_detailed";
 	const char *refname_nons = strip_namespace(refname);
 	struct object_id peeled;
 
 	if (mark_our_ref(refname_nons, refname, oid))
 		return 0;
 
-	if (capabilities) {
-		struct strbuf symref_info = STRBUF_INIT;
-
-		format_symref_info(&symref_info, cb_data);
-		packet_write(1, "%s %s%c%s%s%s%s%s agent=%s\n",
-			     oid_to_hex(oid), refname_nons,
-			     0, capabilities,
-			     (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?
-				     " allow-tip-sha1-in-want" : "",
-			     (allow_unadvertised_object_request & ALLOW_REACHABLE_SHA1) ?
-				     " allow-reachable-sha1-in-want" : "",
-			     stateless_rpc ? " no-done" : "",
-			     symref_info.buf,
-			     git_user_agent_sanitized());
-		strbuf_release(&symref_info);
+	if (advertise_capabilities) {
+		int i;
+		struct strbuf capabilities = STRBUF_INIT;
+
+		for (i = 0; i < ARRAY_SIZE(all_capabilities); i++) {
+			const char *cap = all_capabilities[i];
+			if (!strcmp(cap, "allow-tip-sha1-in-want")
+			    && !(allow_unadvertised_object_request & ALLOW_TIP_SHA1))
+				continue;
+			if (!strcmp(cap, "allow-reachable-sha1-in-want")
+			    && !(allow_unadvertised_object_request & ALLOW_REACHABLE_SHA1))
+				continue;
+			if (!strcmp(cap, "no-done") && !stateless_rpc)
+				continue;
+			strbuf_addf(&capabilities, " %s", cap);
+		}
+
+		format_symref_info(&capabilities, cb_data);
+		strbuf_addf(&capabilities, " agent=%s", git_user_agent_sanitized());
+
+		packet_write(1, "%s %s%c%s\n", oid_to_hex(oid), refname_nons,
+			     0, capabilities.buf);
+		strbuf_release(&capabilities);
+		advertise_capabilities = 0;
 	} else {
 		packet_write(1, "%s %s\n", oid_to_hex(oid), refname_nons);
 	}
-	capabilities = NULL;
 	if (!peel_ref(refname, peeled.hash))
 		packet_write(1, "%s %s^{}\n", oid_to_hex(&peeled), refname_nons);
 	return 0;
-- 
2.8.0.32.g71f8beb.dirty
David Turner· Apr 30, 2016, 01:04 UTC · re: Stefan Beller · lore

Re: [PATCH 02/14] upload-pack.c: Refactor capability advertising

On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
> Instead of having the capabilities in a local string, keep them
> in a struct outside the function. This will allow us in a later patch
nit: s/struct/array/
Junio C Hamano· May 4, 2016, 20:05 UTC · re: Stefan Beller · lore

Re: [PATCH 02/14] upload-pack.c: Refactor capability advertising

Stefan Beller <sbeller@google.com> writes:
Show 6 quoted lines
> Instead of having the capabilities in a local string, keep them
> in a struct outside the function. This will allow us in a later patch
> to easily reuse the capabilities in version 2 of the protocol.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---

Between a flat string and a list, you may be able to enumerate the elements in a list more easily, but my knee-jerk reaction is that this does not go far enough, if you are to introduce a table. It for example does not allow us to lose the conditional writing of some capabilties based on allow_*_request and replace that with a more table-driven approach.

Perhaps that can come in a later step (in other words, the above is not a basis for rejecting this change; just pointing it out that this does not have to be the end of the story).

Show 85 quoted lines
>  upload-pack.c | 59 ++++++++++++++++++++++++++++++++++++++++-------------------
>  1 file changed, 40 insertions(+), 19 deletions(-)
>
> diff --git a/upload-pack.c b/upload-pack.c
> index aaaf883..85381d5 100644
> --- a/upload-pack.c
> +++ b/upload-pack.c
> @@ -719,37 +719,58 @@ static void format_symref_info(struct strbuf *buf, struct string_list *symref)
>  		strbuf_addf(buf, " symref=%s:%s", item->string, (char *)item->util);
>  }
>  
> +static int advertise_capabilities = 1;
> +const char *all_capabilities[] = {
> +	"multi_ack",
> +	"thin-pack",
> +	"side-band",
> +	"side-band-64k",
> +	"ofs-delta",
> +	"shallow",
> +	"no-progress",
> +	"include-tag",
> +	"multi_ack_detailed",
> +	"allow-tip-sha1-in-want",
> +	"allow-reachable-sha1-in-want",
> +	"no-done",
> +};
> +
>  static int send_ref(const char *refname, const struct object_id *oid,
>  		    int flag, void *cb_data)
>  {
> -	static const char *capabilities = "multi_ack thin-pack side-band"
> -		" side-band-64k ofs-delta shallow no-progress"
> -		" include-tag multi_ack_detailed";
>  	const char *refname_nons = strip_namespace(refname);
>  	struct object_id peeled;
>  
>  	if (mark_our_ref(refname_nons, refname, oid))
>  		return 0;
>  
> -	if (capabilities) {
> -		struct strbuf symref_info = STRBUF_INIT;
> -
> -		format_symref_info(&symref_info, cb_data);
> -		packet_write(1, "%s %s%c%s%s%s%s%s agent=%s\n",
> -			     oid_to_hex(oid), refname_nons,
> -			     0, capabilities,
> -			     (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?
> -				     " allow-tip-sha1-in-want" : "",
> -			     (allow_unadvertised_object_request & ALLOW_REACHABLE_SHA1) ?
> -				     " allow-reachable-sha1-in-want" : "",
> -			     stateless_rpc ? " no-done" : "",
> -			     symref_info.buf,
> -			     git_user_agent_sanitized());
> -		strbuf_release(&symref_info);
> +	if (advertise_capabilities) {
> +		int i;
> +		struct strbuf capabilities = STRBUF_INIT;
> +
> +		for (i = 0; i < ARRAY_SIZE(all_capabilities); i++) {
> +			const char *cap = all_capabilities[i];
> +			if (!strcmp(cap, "allow-tip-sha1-in-want")
> +			    && !(allow_unadvertised_object_request & ALLOW_TIP_SHA1))
> +				continue;
> +			if (!strcmp(cap, "allow-reachable-sha1-in-want")
> +			    && !(allow_unadvertised_object_request & ALLOW_REACHABLE_SHA1))
> +				continue;
> +			if (!strcmp(cap, "no-done") && !stateless_rpc)
> +				continue;
> +			strbuf_addf(&capabilities, " %s", cap);
> +		}
> +
> +		format_symref_info(&capabilities, cb_data);
> +		strbuf_addf(&capabilities, " agent=%s", git_user_agent_sanitized());
> +
> +		packet_write(1, "%s %s%c%s\n", oid_to_hex(oid), refname_nons,
> +			     0, capabilities.buf);
> +		strbuf_release(&capabilities);
> +		advertise_capabilities = 0;
>  	} else {
>  		packet_write(1, "%s %s\n", oid_to_hex(oid), refname_nons);
>  	}
> -	capabilities = NULL;
>  	if (!peel_ref(refname, peeled.hash))
>  		packet_write(1, "%s %s^{}\n", oid_to_hex(&peeled), refname_nons);
>  	return 0;
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 03/14] upload-pack-2: Implement the version 2 of upload-pack

In upload-pack-2 we send each capability in its own packet buffer. The construction of upload-pack-2 is a bit unfortunate as I would like it to not be depending on a symlink linking to upload-pack.c, but I did not find another easy way to do it. I would like it to generate upload-pack-2.o from upload-pack.c but with '-DTRANSPORT_VERSION=2' set.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 .gitignore      |  1 +
 Makefile        |  4 ++++
 upload-pack-2.c |  1 +
 upload-pack.c   | 49 ++++++++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 54 insertions(+), 1 deletion(-)
 create mode 120000 upload-pack-2.c
Show changes to 4 files +54 −1

.gitignore, Makefile, upload-pack-2.c, upload-pack.c

diff --git a/.gitignore b/.gitignore
index 5087ce1..9a6ea8c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -166,6 +166,7 @@
 /git-update-server-info
 /git-upload-archive
 /git-upload-pack
+/git-upload-pack-2
 /git-var
 /git-verify-commit
 /git-verify-pack
diff --git a/Makefile b/Makefile
index a83e322..3d96126 100644
--- a/Makefile
+++ b/Makefile
@@ -580,6 +580,7 @@ PROGRAM_OBJS += sh-i18n--envsubst.o
 PROGRAM_OBJS += shell.o
 PROGRAM_OBJS += show-index.o
 PROGRAM_OBJS += upload-pack.o
+PROGRAM_OBJS += upload-pack-2.o
 PROGRAM_OBJS += remote-testsvn.o
 
 # Binary suffix, set to .exe for Windows builds
@@ -648,6 +649,7 @@ OTHER_PROGRAMS = git$X
 # what test wrappers are needed and 'install' will install, in bindir
 BINDIR_PROGRAMS_NEED_X += git
 BINDIR_PROGRAMS_NEED_X += git-upload-pack
+BINDIR_PROGRAMS_NEED_X += git-upload-pack-2
 BINDIR_PROGRAMS_NEED_X += git-receive-pack
 BINDIR_PROGRAMS_NEED_X += git-upload-archive
 BINDIR_PROGRAMS_NEED_X += git-shell
@@ -1730,6 +1732,8 @@ $(BUILT_INS): git$X
 	ln -s $< $@ 2>/dev/null || \
 	cp $< $@
 
+$X%-2.o: EXTRA_CPPFLAGS = '-DTRANSPORT_VERSION=2'
+
 common-cmds.h: generate-cmdlist.sh command-list.txt
 
 common-cmds.h: $(wildcard Documentation/git-*.txt)
diff --git a/upload-pack-2.c b/upload-pack-2.c
new file mode 120000
index 0000000..e30a871
--- /dev/null
+++ b/upload-pack-2.c
@@ -0,0 +1 @@
+upload-pack.c
\ No newline at end of file
diff --git a/upload-pack.c b/upload-pack.c
index 85381d5..edfd417 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -820,6 +820,45 @@ static void upload_pack(void)
 	}
 }
 
+#if (TRANSPORT_VERSION == 2)
+static void send_capabilities_version_2(void)
+{
+	int i;
+	for (i = 0; i < ARRAY_SIZE(all_capabilities); i++) {
+		const char *cap = all_capabilities[i];
+		if (!strcmp(cap, "allow-tip-sha1-in-want")
+		    && !(allow_unadvertised_object_request & ALLOW_TIP_SHA1))
+			continue;
+		if (!strcmp(cap, "no-done") && !stateless_rpc)
+			continue;
+		packet_write(1, "%s\n", cap);
+	}
+
+	packet_write(1, "agent=%s\n", git_user_agent_sanitized());
+	packet_flush(1);
+}
+
+static void receive_capabilities_version_2(void)
+{
+	char *line = packet_read_line(0, NULL);
+	while (line) {
+		parse_features(line);
+		line = packet_read_line(0, NULL);
+	}
+}
+
+static void upload_pack_version_2(void)
+{
+	send_capabilities_version_2();
+	receive_capabilities_version_2();
+
+	/* The rest of the protocol stays the same, capabilities advertising
+	   is disabled though. */
+	advertise_capabilities = 0;
+	upload_pack();
+}
+#endif
+
 static int upload_pack_config(const char *var, const char *value, void *unused)
 {
 	if (!strcmp("uploadpack.allowtipsha1inwant", var)) {
@@ -847,8 +886,11 @@ int main(int argc, char **argv)
 	int strict = 0;
 
 	git_setup_gettext();
-
+#if TRANSPORT_VERSION == 2
+	packet_trace_identity("upload-pack-2");
+#else
 	packet_trace_identity("upload-pack");
+#endif
 	git_extract_argv0_path(argv[0]);
 	check_replace_refs = 0;
 
@@ -891,6 +933,11 @@ int main(int argc, char **argv)
 		die("'%s' does not appear to be a git repository", dir);
 
 	git_config(upload_pack_config, NULL);
+
+#if TRANSPORT_VERSION == 2
+	upload_pack_version_2();
+#else
 	upload_pack();
+#endif
 	return 0;
 }
-- 
2.8.0.32.g71f8beb.dirty
David Turner· May 2, 2016, 17:43 UTC · re: Stefan Beller · lore

Re: [PATCH 03/14] upload-pack-2: Implement the version 2 of upload-pack

On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
Show 8 quoted lines
> In upload-pack-2 we send each capability in its own packet buffer.
> The construction of upload-pack-2 is a bit unfortunate as I would
> like
> it to not be depending on a symlink linking to upload-pack.c, but I
> did
> not find another easy way to do it. I would like it to generate
> upload-pack-2.o from upload-pack.c but with '-DTRANSPORT_VERSION=2'
> set.

Couldn't we check argv[0] and use that to determine protocol? Then we could symlink executables rather than source code.

Stefan Beller· May 2, 2016, 17:51 UTC · re: David Turner · lore

Re: [PATCH 03/14] upload-pack-2: Implement the version 2 of upload-pack

On Mon, May 2, 2016 at 10:43 AM, David Turner <dturner@twopensource.com> wrote:
Show 12 quoted lines
> On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
>> In upload-pack-2 we send each capability in its own packet buffer.
>> The construction of upload-pack-2 is a bit unfortunate as I would
>> like
>> it to not be depending on a symlink linking to upload-pack.c, but I
>> did
>> not find another easy way to do it. I would like it to generate
>> upload-pack-2.o from upload-pack.c but with '-DTRANSPORT_VERSION=2'
>> set.
>
> Couldn't we check argv[0] and use that to determine protocol?  Then we
> could symlink executables rather than source code.
IIRC I proposed a similar thing earlier, i.e.
    if (argv[0] ends with 2)
        do_protocol_v_2(...)

but that may break (and confuse a lot!) some use cases. `git fetch` has the documented --upload-pack switch, so as a server-admin you are free to have git-upload-pack linking to "git-upload-pack-2.8" but additionally you still have "git-upload-pack-1.7" or "git-upload-pack-custom-2".

so I think we should not do that :( I do like to symlink the executables though.

David Turner· May 2, 2016, 18:56 UTC · re: Stefan Beller · lore

Re: [PATCH 03/14] upload-pack-2: Implement the version 2 of upload-pack

On Mon, 2016-05-02 at 10:51 -0700, Stefan Beller wrote:
Show 33 quoted lines
> On Mon, May 2, 2016 at 10:43 AM, David Turner <
> dturner@twopensource.com> wrote:
> > On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
> > > In upload-pack-2 we send each capability in its own packet
> > > buffer.
> > > The construction of upload-pack-2 is a bit unfortunate as I would
> > > like
> > > it to not be depending on a symlink linking to upload-pack.c, but
> > > I
> > > did
> > > not find another easy way to do it. I would like it to generate
> > > upload-pack-2.o from upload-pack.c but with '
> > > -DTRANSPORT_VERSION=2'
> > > set.
> > 
> > Couldn't we check argv[0] and use that to determine protocol?  Then
> > we
> > could symlink executables rather than source code.
> 
> IIRC I proposed a similar thing earlier, i.e.
> 
>     if (argv[0] ends with 2)
>         do_protocol_v_2(...)
> 
> but that may break (and confuse a lot!) some use cases.
> `git fetch` has the documented --upload-pack switch, so as a server
> -admin
> you are free to have git-upload-pack linking to
> "git-upload-pack-2.8" but additionally you still have
> "git-upload-pack-1.7" or "git-upload-pack-custom-2".
> 
> so I think we should not do that :(
> I do like to symlink the executables though.

I think it would probably not break anyone if the new executable name were sufficiently distinctive -- e.g. starts_with (strrchr(argv[0], '/'), "git-upload-pack-protocol-v2"). But it would make custom executables a bit more complicated for the future.

I guess it is better to have silly source code but clean binaries than clean source code but silly user-visible rules.

Duy Nguyen· May 3, 2016, 00:31 UTC · re: David Turner · lore

Re: [PATCH 03/14] upload-pack-2: Implement the version 2 of upload-pack

On Tue, May 3, 2016 at 1:56 AM, David Turner <dturner@twopensource.com> wrote:
Show 42 quoted lines
> On Mon, 2016-05-02 at 10:51 -0700, Stefan Beller wrote:
>> On Mon, May 2, 2016 at 10:43 AM, David Turner <
>> dturner@twopensource.com> wrote:
>> > On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
>> > > In upload-pack-2 we send each capability in its own packet
>> > > buffer.
>> > > The construction of upload-pack-2 is a bit unfortunate as I would
>> > > like
>> > > it to not be depending on a symlink linking to upload-pack.c, but
>> > > I
>> > > did
>> > > not find another easy way to do it. I would like it to generate
>> > > upload-pack-2.o from upload-pack.c but with '
>> > > -DTRANSPORT_VERSION=2'
>> > > set.
>> >
>> > Couldn't we check argv[0] and use that to determine protocol?  Then
>> > we
>> > could symlink executables rather than source code.
>>
>> IIRC I proposed a similar thing earlier, i.e.
>>
>>     if (argv[0] ends with 2)
>>         do_protocol_v_2(...)
>>
>> but that may break (and confuse a lot!) some use cases.
>> `git fetch` has the documented --upload-pack switch, so as a server
>> -admin
>> you are free to have git-upload-pack linking to
>> "git-upload-pack-2.8" but additionally you still have
>> "git-upload-pack-1.7" or "git-upload-pack-custom-2".
>>
>> so I think we should not do that :(
>> I do like to symlink the executables though.
>
> I think it would probably not break anyone if the new executable name
> were sufficiently distinctive -- e.g. starts_with (strrchr(argv[0],
> '/'), "git-upload-pack-protocol-v2"). But it would make custom
> executables a bit more complicated for the future.
>
> I guess it is better to have silly source code but clean binaries than
> clean source code but silly user-visible rules.

Maybe add --version to upload-pack? Then we can have a script git-upload-pack-v2 that does "exec git upload-pack --version=2"

-- 
Duy
Junio C Hamano· May 4, 2016, 20:11 UTC · re: David Turner · lore

Re: [PATCH 03/14] upload-pack-2: Implement the version 2 of upload-pack

David Turner <dturner@twopensource.com> writes:
Show 12 quoted lines
> On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
>> In upload-pack-2 we send each capability in its own packet buffer.
>> The construction of upload-pack-2 is a bit unfortunate as I would
>> like
>> it to not be depending on a symlink linking to upload-pack.c, but I
>> did
>> not find another easy way to do it. I would like it to generate
>> upload-pack-2.o from upload-pack.c but with '-DTRANSPORT_VERSION=2'
>> set.
>
> Couldn't we check argv[0] and use that to determine protocol?  Then we
> could symlink executables rather than source code.

Yeah, I do not have a good suggestion on the mechanism to actually switch between behaviours other than what was already been discussed in the thread, but the code resulting from the patch proposed is too ugly with full of #ifdef for it to be the final form.

Just to make sure nobody gets me wrong; it is OK as a POC to move the discussion forward.

A production quality implemention however would need to either be a single executable that switches behaviour at runtime, or two executables, each with its own *.c file with its own main(), that share code in another common *.c file, I would think.

Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 04/14] connect: rewrite feature parsing to work on string_list

Later on when we introduce the version 2 transport protocol, the capabilities will not be transported in one lone string but each capability will be carried in its own pkt line.

To reuse existing infrastructure we would either need to join the capabilities into a single string again later or refactor the current capability parsing to be using a data structure which fits both versions of the transport protocol. We chose to implement the later.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/receive-pack.c | 15 ++++++---
 connect.c              | 82 +++++++++++++++++++++++---------------------------
 connect.h              |  2 +-
 upload-pack.c          | 13 ++++++--
 4 files changed, 58 insertions(+), 54 deletions(-)
Show changes to 4 files +58 −54

builtin/receive-pack.c, connect.c, connect.h, upload-pack.c

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index a744437..4e98514 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1429,16 +1429,21 @@ static struct command *read_head_info(struct sha1_array *shallow)
 
 		linelen = strlen(line);
 		if (linelen < len) {
-			const char *feature_list = line + linelen + 1;
-			if (parse_feature_request(feature_list, "report-status"))
+			struct string_list feature_list = STRING_LIST_INIT_DUP;
+			string_list_split(&feature_list,
+					  line + linelen + 1,
+					  ' ', -1);
+			if (parse_feature_request(&feature_list, "report-status"))
 				report_status = 1;
-			if (parse_feature_request(feature_list, "side-band-64k"))
+			if (parse_feature_request(&feature_list, "side-band-64k"))
 				use_sideband = LARGE_PACKET_MAX;
-			if (parse_feature_request(feature_list, "quiet"))
+			if (parse_feature_request(&feature_list, "quiet"))
 				quiet = 1;
 			if (advertise_atomic_push
-			    && parse_feature_request(feature_list, "atomic"))
+			    && parse_feature_request(&feature_list, "atomic"))
 				use_atomic = 1;
+
+			string_list_clear(&feature_list, 1);
 		}
 
 		if (!strcmp(line, "push-cert")) {
diff --git a/connect.c b/connect.c
index c53f3f1..79505fb 100644
--- a/connect.c
+++ b/connect.c
@@ -11,8 +11,8 @@
 #include "sha1-array.h"
 #include "transport.h"
 
-static char *server_capabilities;
-static const char *parse_feature_value(const char *, const char *, int *);
+struct string_list server_capabilities = STRING_LIST_INIT_DUP;
+static const char *parse_feature_value(struct string_list *, const char *, int *);
 
 static int check_ref(const char *name, unsigned int flags)
 {
@@ -81,18 +81,18 @@ reject:
 
 static void annotate_refs_with_symref_info(struct ref *ref)
 {
+	struct string_list_item *item;
 	struct string_list symref = STRING_LIST_INIT_DUP;
-	const char *feature_list = server_capabilities;
 
-	while (feature_list) {
-		int len;
+	for_each_string_list_item(item, &server_capabilities) {
 		const char *val;
 
-		val = parse_feature_value(feature_list, "symref", &len);
-		if (!val)
-			break;
-		parse_one_symref_info(&symref, val, len);
-		feature_list = val + 1;
+		if (skip_prefix(item->string, "symref", &val)) {
+			if (!val)
+				continue;
+			val++; /* skip the = */
+			parse_one_symref_info(&symref, val, strlen(val));
+		}
 	}
 	string_list_sort(&symref);
 
@@ -155,9 +155,12 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 		name = buffer + GIT_SHA1_HEXSZ + 1;
 
 		name_len = strlen(name);
+
 		if (len != name_len + GIT_SHA1_HEXSZ + 1) {
-			free(server_capabilities);
-			server_capabilities = xstrdup(name + name_len + 1);
+			string_list_clear(&server_capabilities, 1);
+			string_list_split(&server_capabilities,
+					  name + name_len + 1,
+					  ' ', -1);
 		}
 
 		if (extra_have && !strcmp(name, ".have")) {
@@ -179,51 +182,40 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 	return list;
 }
 
-static const char *parse_feature_value(const char *feature_list, const char *feature, int *lenp)
+static const char *parse_feature_value(struct string_list *feature_list, const char *feature, int *lenp)
 {
-	int len;
-
-	if (!feature_list)
-		return NULL;
-
-	len = strlen(feature);
-	while (*feature_list) {
-		const char *found = strstr(feature_list, feature);
-		if (!found)
-			return NULL;
-		if (feature_list == found || isspace(found[-1])) {
-			const char *value = found + len;
-			/* feature with no value (e.g., "thin-pack") */
-			if (!*value || isspace(*value)) {
-				if (lenp)
-					*lenp = 0;
-				return value;
-			}
-			/* feature with a value (e.g., "agent=git/1.2.3") */
-			else if (*value == '=') {
-				value++;
-				if (lenp)
-					*lenp = strcspn(value, " \t\n");
-				return value;
-			}
-			/*
-			 * otherwise we matched a substring of another feature;
-			 * keep looking
-			 */
+	const char *value;
+	struct string_list_item *item;
+
+	for_each_string_list_item(item, feature_list) {
+		if (!skip_prefix(item->string, feature, &value))
+			continue;
+
+		/* feature with no value (e.g., "thin-pack") */
+		if (!*value) {
+			if (lenp)
+				*lenp = 0;
+			return value;
+		}
+		/* feature with a value (e.g., "agent=git/1.2.3") */
+		else if (*value == '=') {
+			value++;
+			if (lenp)
+				*lenp = strlen(value);
+			return value;
 		}
-		feature_list = found + 1;
 	}
 	return NULL;
 }
 
-int parse_feature_request(const char *feature_list, const char *feature)
+int parse_feature_request(struct string_list *feature_list, const char *feature)
 {
 	return !!parse_feature_value(feature_list, feature, NULL);
 }
 
 const char *server_feature_value(const char *feature, int *len)
 {
-	return parse_feature_value(server_capabilities, feature, len);
+	return parse_feature_value(&server_capabilities, feature, len);
 }
 
 int server_supports(const char *feature)
diff --git a/connect.h b/connect.h
index 01f14cd..eaf21c2 100644
--- a/connect.h
+++ b/connect.h
@@ -9,7 +9,7 @@ extern struct child_process *git_connect(int fd[2], const char *url, const char
 extern int finish_connect(struct child_process *conn);
 extern int git_connection_is_socket(struct child_process *conn);
 extern int server_supports(const char *feature);
-extern int parse_feature_request(const char *features, const char *feature);
+extern int parse_feature_request(struct string_list *, const char *feature);
 extern const char *server_feature_value(const char *feature, int *len_ret);
 extern int url_is_local_not_ssh(const char *url);
 
diff --git a/upload-pack.c b/upload-pack.c
index edfd417..4e69b8e 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -535,7 +535,7 @@ error:
 	}
 }
 
-static void parse_features(const char *features)
+static void parse_features(struct string_list *features)
 {
 	if (parse_feature_request(features, "multi_ack_detailed"))
 		multi_ack = 2;
@@ -567,7 +567,9 @@ static void receive_needs(void)
 	for (;;) {
 		struct object *o;
 		unsigned char sha1_buf[20];
+		struct string_list list = STRING_LIST_INIT_DUP;
 		char *line = packet_read_line(0, NULL);
+
 		reset_timeout();
 		if (!line)
 			break;
@@ -600,7 +602,9 @@ static void receive_needs(void)
 			die("git upload-pack: protocol error, "
 			    "expected to get sha, not '%s'", line);
 
-		parse_features(line + 45);
+		string_list_split(&list, line + 45, ' ', -1);
+		parse_features(&list);
+		string_list_clear(&list, 1);
 
 		o = parse_object(sha1_buf);
 		if (!o)
@@ -840,9 +844,12 @@ static void send_capabilities_version_2(void)
 
 static void receive_capabilities_version_2(void)
 {
+	struct string_list list = STRING_LIST_INIT_NODUP;
 	char *line = packet_read_line(0, NULL);
 	while (line) {
-		parse_features(line);
+		string_list_append(&list, line);
+		parse_features(&list);
+		string_list_clear(&list, 1);
 		line = packet_read_line(0, NULL);
 	}
 }
-- 
2.8.0.32.g71f8beb.dirty
David Turner· May 2, 2016, 18:18 UTC · re: Stefan Beller · lore

Re: [PATCH 04/14] connect: rewrite feature parsing to work on string_list

On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
> +		if (skip_prefix(item->string, "symref", &val)) {
> +			if (!val)
> +				continue;

This if should never happen (skip_prefix returns 0 in that case). You probably meant !*val -- but:

> +			val++; /* skip the = */

I think you should instead skip_prefix "symref=" because: (a) it saves some code. (b) it allows for capabilities like symref_foo to later be added.

> +	struct string_list list = STRING_LIST_INIT_NODUP;
Maybe move the scope of list into the while loop below?
Show 7 quoted lines
>  	char *line = packet_read_line(0, NULL);
>  	while (line) {
> -		parse_features(line);
> +		string_list_append(&list, line);
> +		parse_features(&list);
> +		string_list_clear(&list, 1);
>  		line = packet_read_line(0, NULL);

This is a bit convoluted in the one-feature-per-line case, but I guess I understand that for the sake of generality it's useful.

Stefan Beller· May 2, 2016, 18:46 UTC · re: David Turner · lore

Re: [PATCH 04/14] connect: rewrite feature parsing to work on string_list

On Mon, May 2, 2016 at 11:18 AM, David Turner <dturner@twopensource.com> wrote:
Show 28 quoted lines
> On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
>> +             if (skip_prefix(item->string, "symref", &val)) {
>> +                     if (!val)
>> +                             continue;
>
> This if should never happen (skip_prefix returns 0 in that case).  You
> probably meant !*val -- but:
>
>> +                     val++; /* skip the = */
>
> I think you should instead skip_prefix "symref=" because:
> (a) it saves some code.
> (b) it allows for capabilities like symref_foo to later be added.
>
>> +     struct string_list list = STRING_LIST_INIT_NODUP;
>
> Maybe move the scope of list into the while loop below?
>
>>       char *line = packet_read_line(0, NULL);
>>       while (line) {
>> -             parse_features(line);
>> +             string_list_append(&list, line);
>> +             parse_features(&list);
>> +             string_list_clear(&list, 1);
>>               line = packet_read_line(0, NULL);
>
> This is a bit convoluted in the one-feature-per-line case, but I guess
> I understand that for the sake of generality it's useful.

Thanks for the review, Stefan

Junio C Hamano· May 4, 2016, 20:13 UTC · re: Stefan Beller · lore

Re: [PATCH 04/14] connect: rewrite feature parsing to work on string_list

Stefan Beller <sbeller@google.com> writes:
> Later on when we introduce the version 2 transport protocol, the
> capabilities will not be transported in one lone string but each
s/lone/long/, I think.
Show 14 quoted lines
> capability will be carried in its own pkt line.
>
> To reuse existing infrastructure we would either need to join the
> capabilities into a single string again later or refactor the current
> capability parsing to be using a data structure which fits both
> versions of the transport protocol. We chose to implement the later.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  builtin/receive-pack.c | 15 ++++++---
>  connect.c              | 82 +++++++++++++++++++++++---------------------------
>  connect.h              |  2 +-
>  upload-pack.c          | 13 ++++++--
>  4 files changed, 58 insertions(+), 54 deletions(-)
I am not sure if the churn is make a right tradeoff here.

A loop to concatenate each segment into a strbuf before passing it to parse_feature_request would be at most 5 lines long, no?

David Turner· May 17, 2016, 22:23 UTC · re: Junio C Hamano · lore

Re: [PATCH 04/14] connect: rewrite feature parsing to work on string_list

On Wed, 2016-05-04 at 13:13 -0700, Junio C Hamano wrote:
Show 29 quoted lines
> Stefan Beller <sbeller@google.com> writes:
> 
> > Later on when we introduce the version 2 transport protocol, the
> > capabilities will not be transported in one lone string but each
> 
> s/lone/long/, I think.
> 
> > capability will be carried in its own pkt line.
> > 
> > To reuse existing infrastructure we would either need to join the
> > capabilities into a single string again later or refactor the
> > current
> > capability parsing to be using a data structure which fits both
> > versions of the transport protocol. We chose to implement the
> > later.
> > 
> > Signed-off-by: Stefan Beller <sbeller@google.com>
> > ---
> >  builtin/receive-pack.c | 15 ++++++---
> >  connect.c              | 82 +++++++++++++++++++++++---------------
> > ------------
> >  connect.h              |  2 +-
> >  upload-pack.c          | 13 ++++++--
> >  4 files changed, 58 insertions(+), 54 deletions(-)
> 
> I am not sure if the churn is make a right tradeoff here.
> 
> A loop to concatenate each segment into a strbuf before passing it
> to parse_feature_request would be at most 5 lines long, no?

I started looking at this again briefly today, and I actually think that string manipulation of this sort is a mistake. String manipulation is error-prone, and having an interface defined in terms of strings makes it hard for a reader to see what sorts of input are valid and invalid. (It's also often somewhat less efficient). So I think it is a good idea to improve the interface while we're already changing code in this area.

Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 05/14] transport: add infrastructure to support a protocol version number

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 fetch-pack.h       |  1 +
 remote.c           |  2 ++
 remote.h           |  2 ++
 transport-helper.c |  1 +
 transport.c        | 20 ++++++++++++++++++--
 transport.h        |  8 ++++++++
 6 files changed, 32 insertions(+), 2 deletions(-)
Show changes to 6 files +32 −2

fetch-pack.h, remote.c, remote.h, transport-helper.c, transport.c, transport.h

diff --git a/fetch-pack.h b/fetch-pack.h
index bb7fd76..3314362 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -8,6 +8,7 @@ struct sha1_array;
 
 struct fetch_pack_args {
 	const char *uploadpack;
+	int transport_version;
 	int unpacklimit;
 	int depth;
 	unsigned quiet:1;
diff --git a/remote.c b/remote.c
index 28fd676..760611d 100644
--- a/remote.c
+++ b/remote.c
@@ -9,6 +9,7 @@
 #include "string-list.h"
 #include "mergesort.h"
 #include "argv-array.h"
+#include "transport.h"
 
 enum map_direction { FROM_SRC, FROM_DST };
 
@@ -164,6 +165,7 @@ static struct remote *make_remote(const char *name, int len)
 		return ret;
 
 	ret = xcalloc(1, sizeof(struct remote));
+	ret->transport_version = DEFAULT_TRANSPORT_VERSION;
 	ret->prune = -1;  /* unspecified */
 	ALLOC_GROW(remotes, remotes_nr + 1, remotes_alloc);
 	remotes[remotes_nr++] = ret;
diff --git a/remote.h b/remote.h
index c21fd37..cdb25d0 100644
--- a/remote.h
+++ b/remote.h
@@ -51,6 +51,8 @@ struct remote {
 	const char *receivepack;
 	const char *uploadpack;
 
+	int transport_version;
+
 	/*
 	 * for curl remotes only
 	 */
diff --git a/transport-helper.c b/transport-helper.c
index b934183..3cb386f 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -248,6 +248,7 @@ static int disconnect_helper(struct transport *transport)
 }
 
 static const char *unsupported_options[] = {
+	TRANS_OPT_TRANSPORTVERSION,
 	TRANS_OPT_UPLOADPACK,
 	TRANS_OPT_RECEIVEPACK,
 	TRANS_OPT_THIN,
diff --git a/transport.c b/transport.c
index 095e61f..608b92c 100644
--- a/transport.c
+++ b/transport.c
@@ -151,6 +151,16 @@ static int set_git_option(struct git_transport_options *opts,
 				die("transport: invalid depth option '%s'", value);
 		}
 		return 0;
+	} else if (!strcmp(name, TRANS_OPT_TRANSPORTVERSION)) {
+		if (!value)
+			opts->transport_version = DEFAULT_TRANSPORT_VERSION;
+		else {
+			char *end;
+			opts->transport_version = strtol(value, &end, 0);
+			if (*end)
+				die("transport: invalid transport version option '%s'", value);
+		}
+		return 0;
 	}
 	return 1;
 }
@@ -203,6 +213,7 @@ static int fetch_refs_via_pack(struct transport *transport,
 
 	memset(&args, 0, sizeof(args));
 	args.uploadpack = data->options.uploadpack;
+	args.transport_version = data->options.transport_version;
 	args.keep_pack = data->options.keep;
 	args.lock_pack = 1;
 	args.use_thin_pack = data->options.thin;
@@ -694,6 +705,8 @@ struct transport *transport_get(struct remote *remote, const char *url)
 		ret->connect = connect_git;
 		ret->disconnect = disconnect_git;
 		ret->smart_options = &(data->options);
+		ret->smart_options->transport_version =
+			DEFAULT_TRANSPORT_VERSION;
 
 		data->conn = NULL;
 		data->got_remote_heads = 0;
@@ -706,12 +719,15 @@ struct transport *transport_get(struct remote *remote, const char *url)
 
 	if (ret->smart_options) {
 		ret->smart_options->thin = 1;
-		ret->smart_options->uploadpack = "git-upload-pack";
+		ret->smart_options->uploadpack = DEFAULT_TRANSPORT_UPLOAD_PACK;
 		if (remote->uploadpack)
 			ret->smart_options->uploadpack = remote->uploadpack;
-		ret->smart_options->receivepack = "git-receive-pack";
+		ret->smart_options->receivepack = DEFAULT_TRANSPORT_RECEIVE_PACK;
 		if (remote->receivepack)
 			ret->smart_options->receivepack = remote->receivepack;
+		if (remote->transport_version)
+			ret->smart_options->transport_version =
+				remote->transport_version;
 	}
 
 	return ret;
diff --git a/transport.h b/transport.h
index c681408..af90529 100644
--- a/transport.h
+++ b/transport.h
@@ -13,6 +13,7 @@ struct git_transport_options {
 	unsigned self_contained_and_connected : 1;
 	unsigned update_shallow : 1;
 	int depth;
+	int transport_version;
 	const char *uploadpack;
 	const char *receivepack;
 	struct push_cas_option *cas;
@@ -188,6 +189,13 @@ int transport_restrict_protocols(void);
 /* Send push certificates */
 #define TRANS_OPT_PUSH_CERT "pushcert"
 
+/* Use a new version of the git protocol */
+#define TRANS_OPT_TRANSPORTVERSION "transportversion"
+
+#define DEFAULT_TRANSPORT_VERSION 1
+#define DEFAULT_TRANSPORT_UPLOAD_PACK "git-upload-pack"
+#define DEFAULT_TRANSPORT_RECEIVE_PACK "git-receive-pack"
+
 /**
  * Returns 0 if the option was used, non-zero otherwise. Prints a
  * message to stderr if the option is not used.
-- 
2.8.0.32.g71f8beb.dirty
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 06/14] remote.h: add get_remote_capabilities, request_capabilities

Instead of calling get_remote_heads as a first command during the protocol exchange, we need to have fine grained control over the capability negotiation in version 2 of the protocol.

Introduce get_remote_capabilities, which will just listen to capabilities of the remote and request_capabilities which will tell the selection of capabilities to the remote.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 connect.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 remote.h  |  3 +++
 2 files changed, 62 insertions(+)
Show changes to 2 files +62 −0

connect.c, remote.h

diff --git a/connect.c b/connect.c
index 79505fb..1ba9a0f 100644
--- a/connect.c
+++ b/connect.c
@@ -10,6 +10,7 @@
 #include "string-list.h"
 #include "sha1-array.h"
 #include "transport.h"
+#include "version.h"
 
 struct string_list server_capabilities = STRING_LIST_INIT_DUP;
 static const char *parse_feature_value(struct string_list *, const char *, int *);
@@ -106,6 +107,64 @@ static void annotate_refs_with_symref_info(struct ref *ref)
 	string_list_clear(&symref, 0);
 }
 
+const char *known_capabilities[] = {
+	"multi_ack",
+	"thin-pack",
+	"side-band",
+	"side-band-64k",
+	"ofs-delta",
+	"shallow",
+	"no-progress",
+	"include-tag",
+	"multi_ack_detailed",
+	"allow-tip-sha1-in-want",
+	"allow-reachable-sha1-in-want",
+	"no-done",
+};
+
+static int keep_capability(char *line)
+{
+	int i;
+	for (i = 0; i < ARRAY_SIZE(known_capabilities); i++)
+		if (starts_with(line, known_capabilities[i]))
+			return 1;
+	return 0;
+}
+
+void get_remote_capabilities(int in, char *src_buf, size_t src_len)
+{
+	for (;;) {
+		int len;
+		char *line = packet_buffer;
+
+		len = packet_read(in, &src_buf, &src_len,
+				  packet_buffer, sizeof(packet_buffer),
+				  PACKET_READ_GENTLE_ON_EOF |
+				  PACKET_READ_CHOMP_NEWLINE);
+		if (len < 0)
+			die_initial_contact(0);
+
+		if (!len)
+			break;
+
+		/*
+		 * We need to ignore and drop unknown capabilities as they
+		 * may be huge.
+		 */
+		if (keep_capability(line))
+			string_list_append(&server_capabilities, line);
+	}
+}
+
+
+void request_capabilities(int out, struct string_list *list)
+{
+	struct string_list_item *item;
+	for_each_string_list_item(item, list)
+		packet_write(out, "%s\n", item->string);
+	packet_flush(out);
+}
+
 /*
  * Read all the refs from the other end
  */
diff --git a/remote.h b/remote.h
index cdb25d0..534282b 100644
--- a/remote.h
+++ b/remote.h
@@ -153,6 +153,9 @@ extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,
 				     struct sha1_array *extra_have,
 				     struct sha1_array *shallow);
 
+extern void get_remote_capabilities(int in, char *src_buf, size_t src_len);
+extern void request_capabilities(int out, struct string_list*);
+
 int resolve_remote_symref(struct ref *ref, struct ref *list);
 int ref_newer(const struct object_id *new_oid, const struct object_id *old_oid);
 
-- 
2.8.0.32.g71f8beb.dirty
David Turner· May 2, 2016, 18:57 UTC · re: Stefan Beller · lore

Re: [PATCH 06/14] remote.h: add get_remote_capabilities, request_capabilities

On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
 
Show 14 quoted lines
> +const char *known_capabilities[] = {
> +	"multi_ack",
> +	"thin-pack",
> +	"side-band",
> +	"side-band-64k",
> +	"ofs-delta",
> +	"shallow",
> +	"no-progress",
> +	"include-tag",
> +	"multi_ack_detailed",
> +	"allow-tip-sha1-in-want",
> +	"allow-reachable-sha1-in-want",
> +	"no-done",
> +};

I wonder if it is possible to not repeat the list from upload-pack.c? It seems unfortunate to have to add the same string in two places whenever you add a capability.

> +static int keep_capability(char *line)

s/keep_/is_known_/ ? Also it would be good to handle capabilities that are prefixes of others correctly.

Show 9 quoted lines
> +{
> +	int i;
> +	for (i = 0; i < ARRAY_SIZE(known_capabilities); i++)
> +		if (starts_with(line, known_capabilities[i]))
> +			return 1;
> +	return 0;
> +}
> +
> +void get_remote_capabilities(int in, char *src_buf, size_t src_len)

maybe rename "in" to "fd" or "in_fd"? I don't immediately know what "in" is supposed to be when I just look at this signature.

> +void request_capabilities(int out, struct string_list *list)
Maybe name this "send_capabilities_request"?
Jeff King· May 3, 2016, 05:33 UTC · re: David Turner · lore

Re: [PATCH 06/14] remote.h: add get_remote_capabilities, request_capabilities

On Mon, May 02, 2016 at 02:57:43PM -0400, David Turner wrote:
Show 20 quoted lines
> On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
>  
> > +const char *known_capabilities[] = {
> > +	"multi_ack",
> > +	"thin-pack",
> > +	"side-band",
> > +	"side-band-64k",
> > +	"ofs-delta",
> > +	"shallow",
> > +	"no-progress",
> > +	"include-tag",
> > +	"multi_ack_detailed",
> > +	"allow-tip-sha1-in-want",
> > +	"allow-reachable-sha1-in-want",
> > +	"no-done",
> > +};
> 
> I wonder if it is possible to not repeat the list from upload-pack.c?
> It seems unfortunate to have to add the same string in two places
> whenever you add a capability.

I think that in general, we'd stop adding capabilities to v1. If you have a client which speaks the new capability, then it should also be speaking the new protocol. That's not strictly true if other non-git.git implementations want to learn capability X but not protocol v2, but I think in practice it's not an unreasonable world view.

I guess there may be a grey area for a while, though, where even v2-capable clients don't end up speaking it, because they don't yet know that a particular server can handle it. So any capabilities added in that grey area may want to go to both v1 and v2.

-Peff
David Turner· May 3, 2016, 21:21 UTC · re: Jeff King · lore

Re: [PATCH 06/14] remote.h: add get_remote_capabilities, request_capabilities

On Tue, 2016-05-03 at 01:33 -0400, Jeff King wrote:
Show 36 quoted lines
> On Mon, May 02, 2016 at 02:57:43PM -0400, David Turner wrote:
> 
> > On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
> >  
> > > +const char *known_capabilities[] = {
> > > +	"multi_ack",
> > > +	"thin-pack",
> > > +	"side-band",
> > > +	"side-band-64k",
> > > +	"ofs-delta",
> > > +	"shallow",
> > > +	"no-progress",
> > > +	"include-tag",
> > > +	"multi_ack_detailed",
> > > +	"allow-tip-sha1-in-want",
> > > +	"allow-reachable-sha1-in-want",
> > > +	"no-done",
> > > +};
> > 
> > I wonder if it is possible to not repeat the list from upload
> > -pack.c?
> > It seems unfortunate to have to add the same string in two places
> > whenever you add a capability.
> 
> I think that in general, we'd stop adding capabilities to v1. If you
> have a client which speaks the new capability, then it should also be
> speaking the new protocol. That's not strictly true if other non
> -git.git
> implementations want to learn capability X but not protocol v2, but I
> think in practice it's not an unreasonable world view.
> 
> I guess there may be a grey area for a while, though, where even
> v2-capable clients don't end up speaking it, because they don't yet
> know
> that a particular server can handle it. So any capabilities added in
> that grey area may want to go to both v1 and v2.

OK, but then there should be one list per protocol version rather than two copies of the same list.

Stefan Beller· May 4, 2016, 16:44 UTC · re: David Turner · lore

Re: [PATCH 06/14] remote.h: add get_remote_capabilities, request_capabilities

On Tue, May 3, 2016 at 2:21 PM, David Turner <dturner@twopensource.com> wrote:
Show 41 quoted lines
> On Tue, 2016-05-03 at 01:33 -0400, Jeff King wrote:
>> On Mon, May 02, 2016 at 02:57:43PM -0400, David Turner wrote:
>>
>> > On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
>> >
>> > > +const char *known_capabilities[] = {
>> > > + "multi_ack",
>> > > + "thin-pack",
>> > > + "side-band",
>> > > + "side-band-64k",
>> > > + "ofs-delta",
>> > > + "shallow",
>> > > + "no-progress",
>> > > + "include-tag",
>> > > + "multi_ack_detailed",
>> > > + "allow-tip-sha1-in-want",
>> > > + "allow-reachable-sha1-in-want",
>> > > + "no-done",
>> > > +};
>> >
>> > I wonder if it is possible to not repeat the list from upload
>> > -pack.c?
>> > It seems unfortunate to have to add the same string in two places
>> > whenever you add a capability.
>>
>> I think that in general, we'd stop adding capabilities to v1. If you
>> have a client which speaks the new capability, then it should also be
>> speaking the new protocol. That's not strictly true if other non
>> -git.git
>> implementations want to learn capability X but not protocol v2, but I
>> think in practice it's not an unreasonable world view.
>>
>> I guess there may be a grey area for a while, though, where even
>> v2-capable clients don't end up speaking it, because they don't yet
>> know
>> that a particular server can handle it. So any capabilities added in
>> that grey area may want to go to both v1 and v2.
>
> OK, but then there should be one list per protocol version rather than
> two copies of the same list.
>

I thought this is by design as upload-pack is a different program, i.e. it could be developed out of sync with the client, adding/removing capabilities there but not in fetch-pack. That doesn't make sense though.

We could introduce known_capabilities_v1 and _v2 respectively in shared header files, though.

Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 07/14] fetch-pack: move capability selection out of do_fetch_pack

Later in version 2 of the pack protocol the selection of capabilities happens at another step of the protocol, so move out the current capability selection, so we can reuse it later more easily.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 fetch-pack.c | 28 +++++++++++++++++-----------
 1 file changed, 17 insertions(+), 11 deletions(-)
Show changes to fetch-pack.c +17 −11
diff --git a/fetch-pack.c b/fetch-pack.c
index f96f6df..53f6384 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -796,21 +796,11 @@ static int cmp_ref_by_name(const void *a_, const void *b_)
 	return strcmp(a->name, b->name);
 }
 
-static struct ref *do_fetch_pack(struct fetch_pack_args *args,
-				 int fd[2],
-				 const struct ref *orig_ref,
-				 struct ref **sought, int nr_sought,
-				 struct shallow_info *si,
-				 char **pack_lockfile)
+static void select_capabilities(struct fetch_pack_args *args)
 {
-	struct ref *ref = copy_ref_list(orig_ref);
-	unsigned char sha1[20];
 	const char *agent_feature;
 	int agent_len;
 
-	sort_ref_list(&ref, ref_compare_name);
-	qsort(sought, nr_sought, sizeof(*sought), cmp_ref_by_name);
-
 	if ((args->depth > 0 || is_repository_shallow()) && !server_supports("shallow"))
 		die("Server does not support shallow clients");
 	if (server_supports("multi_ack_detailed")) {
@@ -867,6 +857,22 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 			fprintf(stderr, "Server version is %.*s\n",
 				agent_len, agent_feature);
 	}
+}
+
+static struct ref *do_fetch_pack(struct fetch_pack_args *args,
+				 int fd[2],
+				 const struct ref *orig_ref,
+				 struct ref **sought, int nr_sought,
+				 struct shallow_info *si,
+				 char **pack_lockfile)
+{
+	struct ref *ref = copy_ref_list(orig_ref);
+	unsigned char sha1[20];
+
+	sort_ref_list(&ref, ref_compare_name);
+	qsort(sought, nr_sought, sizeof(*sought), cmp_ref_by_name);
+
+	select_capabilities(args);
 
 	if (everything_local(args, &ref, sought, nr_sought)) {
 		packet_flush(fd[1]);
-- 
2.8.0.32.g71f8beb.dirty
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 08/14] fetch-pack: factor out get_selected_capabilities_list

This will be used later by both versions of the transport protocol.
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 fetch-pack.c | 51 ++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 40 insertions(+), 11 deletions(-)
Show changes to fetch-pack.c +40 −11
diff --git a/fetch-pack.c b/fetch-pack.c
index 53f6384..b43490f 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -257,6 +257,9 @@ static int next_flush(struct fetch_pack_args *args, int count)
 	return count;
 }
 
+static void get_selected_capabilities_list(struct string_list *list,
+					   struct fetch_pack_args *args);
+
 static int find_common(struct fetch_pack_args *args,
 		       int fd[2], unsigned char *result_sha1,
 		       struct ref *refs)
@@ -302,18 +305,15 @@ static int find_common(struct fetch_pack_args *args,
 
 		remote_hex = sha1_to_hex(remote);
 		if (!fetching) {
+			struct string_list list = STRING_LIST_INIT_NODUP;
 			struct strbuf c = STRBUF_INIT;
-			if (multi_ack == 2)     strbuf_addstr(&c, " multi_ack_detailed");
-			if (multi_ack == 1)     strbuf_addstr(&c, " multi_ack");
-			if (no_done)            strbuf_addstr(&c, " no-done");
-			if (use_sideband == 2)  strbuf_addstr(&c, " side-band-64k");
-			if (use_sideband == 1)  strbuf_addstr(&c, " side-band");
-			if (args->use_thin_pack) strbuf_addstr(&c, " thin-pack");
-			if (args->no_progress)   strbuf_addstr(&c, " no-progress");
-			if (args->include_tag)   strbuf_addstr(&c, " include-tag");
-			if (prefer_ofs_delta)   strbuf_addstr(&c, " ofs-delta");
-			if (agent_supported)    strbuf_addf(&c, " agent=%s",
-							    git_user_agent_sanitized());
+			struct string_list_item *item;
+			get_selected_capabilities_list(&list, args);
+			for_each_string_list_item(item, &list) {
+				strbuf_addstr(&c, " ");
+				strbuf_addstr(&c, item->string);
+			}
+
 			packet_buf_write(&req_buf, "want %s%s\n", remote_hex, c.buf);
 			strbuf_release(&c);
 		} else
@@ -859,6 +859,35 @@ static void select_capabilities(struct fetch_pack_args *args)
 	}
 }
 
+static void get_selected_capabilities_list(struct string_list *list,
+					   struct fetch_pack_args *args)
+{
+	if (multi_ack == 2)
+		string_list_append(list, "multi_ack_detailed");
+	if (multi_ack == 1)
+		string_list_append(list, "multi_ack");
+	if (no_done)
+		string_list_append(list, "no-done");
+	if (use_sideband == 2)
+		string_list_append(list, "side-band-64k");
+	if (use_sideband == 1)
+		string_list_append(list, "side-band");
+	if (args->use_thin_pack)
+		string_list_append(list, "thin-pack");
+	if (args->no_progress)
+		string_list_append(list, "no-progress");
+	if (args->include_tag)
+		string_list_append(list, "include-tag");
+	if (prefer_ofs_delta)
+		string_list_append(list, "ofs-delta");
+	if (agent_supported) {
+		struct strbuf buf = STRBUF_INIT;
+		strbuf_addf(&buf, "agent=%s", git_user_agent_sanitized());
+		string_list_append(list, buf.buf);
+		strbuf_release(&buf);
+	}
+}
+
 static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 				 int fd[2],
 				 const struct ref *orig_ref,
-- 
2.8.0.32.g71f8beb.dirty
David Turner· May 2, 2016, 19:09 UTC · re: Stefan Beller · lore

Re: [PATCH 08/14] fetch-pack: factor out get_selected_capabilities_list

On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
Show 7 quoted lines
> +			struct string_list_item *item;
> +			get_selected_capabilities_list(&list, args);
> +			for_each_string_list_item(item, &list) {
> +				strbuf_addstr(&c, " ");
> +				strbuf_addstr(&c, item->string);
> +			}
> +
I think the contents of list leak here -- you need a string_list_clear.
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 09/14] fetch-pack: Add negotiate_capabilities

As in the pack protocol version 2 we want to negotiate the capabilities before any other exchange we will have a function which will take care of the whole negotiation process.

It will be placed in fetch-pack.c for now as there we have access to its internal variables and we'll work on a `struct fetch_pack_args`. Eventually we want to move it to a better place such as transport.c

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 fetch-pack.c | 14 ++++++++++++++
 fetch-pack.h |  6 ++++++
 2 files changed, 20 insertions(+)
Show changes to 2 files +20 −0

fetch-pack.c, fetch-pack.h

diff --git a/fetch-pack.c b/fetch-pack.c
index b43490f..1544629 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -888,6 +888,20 @@ static void get_selected_capabilities_list(struct string_list *list,
 	}
 }
 
+void negotiate_capabilities(int fd[2], struct fetch_pack_args *args)
+{
+	struct string_list list = STRING_LIST_INIT_NODUP;
+
+	get_remote_capabilities(fd[0], NULL, 0);
+
+	select_capabilities(args);
+	get_selected_capabilities_list(&list, args);
+
+	request_capabilities(fd[1], &list);
+
+	string_list_clear(&list, 1);
+}
+
 static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 				 int fd[2],
 				 const struct ref *orig_ref,
diff --git a/fetch-pack.h b/fetch-pack.h
index 3314362..198498a 100644
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -29,6 +29,12 @@ struct fetch_pack_args {
 };
 
 /*
+ * In version 2 of the pack protocol we negotiate the capabilities
+ * before the actual transfer of refs and packs.
+ */
+void negotiate_capabilities(int fd[2], struct fetch_pack_args *args);
+
+/*
  * sought represents remote references that should be updated from.
  * On return, the names that were found on the remote will have been
  * marked as such.
-- 
2.8.0.32.g71f8beb.dirty
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 10/14] do_fetch_pack: select capabilities for transport version 1 only

`select_capabilities` would set local variables depending on the user selection provided by `args` and the server advertisement which is kept in a list in connect.c

When talking pack protocol version 2 however this has already happend before during 'negotiate_capabilities'

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 fetch-pack.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
Show changes to fetch-pack.c +10 −1
diff --git a/fetch-pack.c b/fetch-pack.c
index 1544629..5ca1e97 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -915,7 +915,16 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
 	sort_ref_list(&ref, ref_compare_name);
 	qsort(sought, nr_sought, sizeof(*sought), cmp_ref_by_name);
 
-	select_capabilities(args);
+	switch (args->transport_version) {
+	case 2:
+		/* capability selection already happend */
+		break;
+	case 1:
+		select_capabilities(args);
+		break;
+	default:
+		die ("transport version %d not supported", args->transport_version);
+	}
 
 	if (everything_local(args, &ref, sought, nr_sought)) {
 		packet_flush(fd[1]);
-- 
2.8.0.32.g71f8beb.dirty
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 11/14] builtin/fetch-pack: add argument for transport version

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/fetch-pack.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
Show changes to builtin/fetch-pack.c +19 −1
diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index bfd0be4..afb614b 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -4,6 +4,7 @@
 #include "remote.h"
 #include "connect.h"
 #include "sha1-array.h"
+#include "transport.h"
 
 static const char fetch_pack_usage[] =
 "git fetch-pack [--all] [--stdin] [--quiet | -q] [--keep | -k] [--thin] "
@@ -56,6 +57,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 
 	memset(&args, 0, sizeof(args));
 	args.uploadpack = "git-upload-pack";
+	args.transport_version = DEFAULT_TRANSPORT_VERSION;
 
 	for (i = 1; i < argc && *argv[i] == '-'; i++) {
 		const char *arg = argv[i];
@@ -130,6 +132,10 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 			args.update_shallow = 1;
 			continue;
 		}
+		if (starts_with(arg, "--transport-version=")) {
+			args.transport_version = strtol(arg + 20, NULL, 0);
+			continue;
+		}
 		usage(fetch_pack_usage);
 	}
 
@@ -178,7 +184,19 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
 		if (!conn)
 			return args.diag_url ? 0 : 1;
 	}
-	get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
+
+	switch (args.transport_version) {
+	case 2: /* first talk about capabilities, then get the refs */
+		negotiate_capabilities(fd, &args);
+		/* fall through */
+	case 1:
+		get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);
+		break;
+	default:
+		die("BUG: Transport version %d not supported",
+			args.transport_version);
+		break;
+	}
 
 	ref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,
 			 &shallow, pack_lockfile_ptr);
-- 
2.8.0.32.g71f8beb.dirty
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 12/14] Add test for fetch-pack

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 t/t5500-fetch-pack.sh | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
Show changes to t/t5500-fetch-pack.sh +21 −0
diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index 91a69fc..2c704ef 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -519,6 +519,7 @@ test_expect_success 'test --all, --depth, and explicit tag' '
 '
 
 test_expect_success 'shallow fetch with tags does not break the repository' '
+	test_when_finished "rm -rf repo1" &&
 	mkdir repo1 &&
 	(
 		cd repo1 &&
@@ -547,6 +548,26 @@ test_expect_success 'fetch-pack can fetch a raw sha1' '
 	git fetch-pack hidden $(git -C hidden rev-parse refs/hidden/one)
 '
 
+test_expect_success 'fetch-pack with protocol version 2' '
+	test_when_finished "rm -rf repo1" &&
+	mkdir repo1 &&
+	(
+		cd repo1 &&
+		git init &&
+		test_commit 1 &&
+		test_commit 2 &&
+		test_commit 3 &&
+		echo "$(git rev-parse master) refs/heads/master" >expected &&
+		mkdir repo2 &&
+		(
+			cd repo2 &&
+			git init &&
+			git fetch-pack --transport-version=2 --upload-pack=git-upload-pack-2 ../.git refs/heads/master >../actual
+		) &&
+		test_cmp expected actual
+	)
+'
+
 check_prot_path () {
 	cat >expected <<-EOF &&
 	Diag: url=$1
-- 
2.8.0.32.g71f8beb.dirty
David Turner· May 2, 2016, 19:45 UTC · re: Stefan Beller · lore

Re: [PATCH 12/14] Add test for fetch-pack

On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
Show 21 quoted lines
> +test_expect_success 'fetch-pack with protocol version 2' '
> +	test_when_finished "rm -rf repo1" &&
> +	mkdir repo1 &&
> +	(
> +		cd repo1 &&
> +		git init &&
> +		test_commit 1 &&
> +		test_commit 2 &&
> +		test_commit 3 &&
> +		echo "$(git rev-parse master) refs/heads/master"
> >expected &&
> +		mkdir repo2 &&
> +		(
> +			cd repo2 &&
> +			git init &&
> +			git fetch-pack --transport-version=2 -
> -upload-pack=git-upload-pack-2 ../.git refs/heads/master >../actual
> +		) &&
> +		test_cmp expected actual
> +	)
> +'

This doesn't actually test that protocol v2 is in fact used (it just tests that --transport-version=2 doesn't crash). It would be nice to actually test the version in-use.

Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 13/14] WIP add test for git pull

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 t/t5520-pull.sh | 6 ++++++
 1 file changed, 6 insertions(+)
Show changes to t/t5520-pull.sh +6 −0
diff --git a/t/t5520-pull.sh b/t/t5520-pull.sh
index 739c089..9bd1feb 100755
--- a/t/t5520-pull.sh
+++ b/t/t5520-pull.sh
@@ -599,4 +599,10 @@ test_expect_success 'git pull --rebase against local branch' '
 	test file = "$(cat file2)"
 '
 
+test_expect_success 'git pull with protocol version 2' '
+	test_pause &&
+	git pull --transport-version=2
+
+'
+
 test_done
-- 
2.8.0.32.g71f8beb.dirty
Stefan Beller· Apr 29, 2016, 23:34 UTC · re: Stefan Beller · lore

[PATCH 14/14] WIP test git fetch

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 t/t5510-fetch.sh | 5 +++++
 1 file changed, 5 insertions(+)
Show changes to t/t5510-fetch.sh +5 −0
diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index 38321d1..ab5d3da 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -687,4 +687,9 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
+test_expect_success 'fetching with transport protocol 2 works' '
+	test_pause
+	git fetch --transport-protocol=2
+'
+
 test_done
-- 
2.8.0.32.g71f8beb.dirty
David Turner· May 2, 2016, 20:41 UTC · re: Stefan Beller · lore

Re: [WIP PATCH 00/14] Protocol v2 patches

On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
Show 18 quoted lines
> Hi David,
> 
> here are my patches for a protocol v2.
> 
> ("Negotiate capabilities before doing anything else", or as code:
> 
>         static void upload_pack_version_2(void)
>         {
>                 send_capabilities_version_2();
>                 receive_capabilities_version_2();
> 
>                 /* The rest of the protocol stays the same,
> capabilities advertising
>                    is disabled though. */
>                 advertise_capabilities = 0;
>                 upload_pack();
>         }
> )
Overall, except for the comments I made, these patches seem sensible. 

Would it be possible to add some docs on the new protocol when you re -roll? (I know these are just the initial patches, but it really helps me to see an explanation along with the code).

Stefan Beller· May 2, 2016, 20:43 UTC · re: David Turner · lore

Re: [WIP PATCH 00/14] Protocol v2 patches

On Mon, May 2, 2016 at 1:41 PM, David Turner <dturner@twopensource.com> wrote:
Show 26 quoted lines
> On Fri, 2016-04-29 at 16:34 -0700, Stefan Beller wrote:
>> Hi David,
>>
>> here are my patches for a protocol v2.
>>
>> ("Negotiate capabilities before doing anything else", or as code:
>>
>>         static void upload_pack_version_2(void)
>>         {
>>                 send_capabilities_version_2();
>>                 receive_capabilities_version_2();
>>
>>                 /* The rest of the protocol stays the same,
>> capabilities advertising
>>                    is disabled though. */
>>                 advertise_capabilities = 0;
>>                 upload_pack();
>>         }
>> )
>
> Overall, except for the comments I made, these patches seem sensible.
>
> Would it be possible to add some docs on the new protocol when you re
> -roll?  (I know these are just the initial patches, but it really helps
> me to see an explanation along with the code).
>

Thanks for the review :) I'll fix the issues and add some docs, though the reroll may take some time. I really want to get the submodule groups stuff done as well.

David Turner· May 24, 2016, 22:46 UTC · re: Stefan Beller · lore

Re: [WIP PATCH 00/14] Protocol v2 patches

I was looking at this again today, and noticed that it doesn't really address the HTTP case.

The central problem is that protocol v2 goes like this: server: I have capabilities w,x,y, and z client: I want capabilities x and z.

But HTTP goes like this: client: [request] server: [response]

I tried to make libcurl do the receive-before-sending thing, but it doesn't seem to be designed for it (even if you prime things by sending a "hello" from the client first). My thought was to hook up CURLOPT_READFUNCTION and CURLOPT_WRITEFUNCTION, and have the read function return CURL_READFUNC_PAUSE and then have the write (=client receiving data ) function unpause the reader (= client sending data) once it gets the capabilities. But apparently pausing only works with chunked encoding, which seems to cause Apache's mod_cgi to fail.

Maybe I'm missing something. Has anyone else ever made something like this work?

Of course, I could always use CURLOPT_CONNECT_ONLY to write my own HTTP client, but that seems pretty unreasonable.

I also looked to see if libcurl had websockets support, since that's one kind of bidirectional conversation over HTTP, but it doesn't seem to.

Another choice is to make a separate /capabilities endpoint that gets
hit before /info/refs.  This is a bit bad because:
(a) it's another HTTP request
(b) it adds implicit state to the HTTP conversation.  If multiple git
servers were behind a load balancer, you might end up getting server A
for /capabilities and server B for /info/refs, and those servers might
have different capabilities.  This is not impossible when testing a git
server upgrade on one machine before rolling it out to a whole fleet. 
 Maybe the rule for clients re capabilities is that they can request
whatever capabilities they want, but the server is free to ignore that
request and send whatever data it feels like.  That's not great, but it
should work (I think).
Does anyone else have any thoughts on how this ought to work?
Duy Nguyen· May 24, 2016, 23:03 UTC · re: David Turner · lore

Re: [WIP PATCH 00/14] Protocol v2 patches

On Wed, May 25, 2016 at 5:46 AM, David Turner <dturner@twopensource.com> wrote:
Show 22 quoted lines
> I was looking at this again today, and noticed that it doesn't really
> address the HTTP case.
>
> The central problem is that protocol v2 goes like this:
> server: I have capabilities w,x,y, and z
> client: I want capabilities x and z.
>
> But HTTP goes like this:
> client: [request]
> server: [response]
>
> I tried to make libcurl do the receive-before-sending thing, but it
> doesn't seem to be designed for it (even if you prime things by sending
> a "hello" from the client first).  My thought was to hook up
> CURLOPT_READFUNCTION and CURLOPT_WRITEFUNCTION, and have the read
> function return CURL_READFUNC_PAUSE and then have the write (=client
> receiving data ) function unpause the reader (= client sending data)
> once it gets the capabilities.  But apparently pausing only works with
> chunked encoding, which seems to cause Apache's mod_cgi to fail.
>
> Maybe I'm missing something.  Has anyone else ever made something like
> this work?
It simply takes one more round-trip to negotiate. Not the best thing, but...
> I also looked to see if libcurl had websockets support, since that's
> one kind of bidirectional conversation over HTTP, but it doesn't seem
> to.

Yeah. And libcurl probably will not support websockets even in long run. I've been searching for a websocket implementation for git and finally settled for netcat-like programs, sitting in front of git and dealing with network just like ssh. It will be the simplest way to add either websocket or http/2 support. If either protocol gets popular enough, smart-http can become a fall-back mechanism where performance does not matter much.

-- 
Duy
David Turner· May 25, 2016, 16:45 UTC · re: Duy Nguyen · lore

Re: [WIP PATCH 00/14] Protocol v2 patches

On Wed, 2016-05-25 at 06:03 +0700, Duy Nguyen wrote:
Show 33 quoted lines
> On Wed, May 25, 2016 at 5:46 AM, David Turner <
> dturner@twopensource.com> wrote:
> > I was looking at this again today, and noticed that it doesn't
> > really
> > address the HTTP case.
> > 
> > The central problem is that protocol v2 goes like this:
> > server: I have capabilities w,x,y, and z
> > client: I want capabilities x and z.
> > 
> > But HTTP goes like this:
> > client: [request]
> > server: [response]
> > 
> > I tried to make libcurl do the receive-before-sending thing, but it
> > doesn't seem to be designed for it (even if you prime things by
> > sending
> > a "hello" from the client first).  My thought was to hook up
> > CURLOPT_READFUNCTION and CURLOPT_WRITEFUNCTION, and have the read
> > function return CURL_READFUNC_PAUSE and then have the write
> > (=client
> > receiving data ) function unpause the reader (= client sending
> > data)
> > once it gets the capabilities.  But apparently pausing only works
> > with
> > chunked encoding, which seems to cause Apache's mod_cgi to fail.
> > 
> > Maybe I'm missing something.  Has anyone else ever made something
> > like
> > this work?
> 
> It simply takes one more round-trip to negotiate. Not the best thing,
> but...

Do you mean that it can be done with libcurl? Or do you mean that I should go with the /capabilities endpoint?

Junio C Hamano· May 25, 2016, 16:23 UTC · re: David Turner · lore

Re: [WIP PATCH 00/14] Protocol v2 patches

David Turner <dturner@twopensource.com> writes:
Show 10 quoted lines
> I was looking at this again today, and noticed that it doesn't really
> address the HTTP case.
>
> The central problem is that protocol v2 goes like this:
> server: I have capabilities w,x,y, and z
> client: I want capabilities x and z.
>
> But HTTP goes like this:
> client: [request]
> server: [response]
I wonder if that can be solved by speculative request?

Let the connection initiator say "If you can do x and z, please do so", and allow the responder to say either "OK, I can do x and z; by the way the full capabilites I support are w, x, y and z", or "Sorry, can't do that; I have capabilities w, x, and y".

David Turner· May 25, 2016, 19:31 UTC · re: Junio C Hamano · lore

Re: [WIP PATCH 00/14] Protocol v2 patches

On Wed, 2016-05-25 at 09:23 -0700, Junio C Hamano wrote:
Show 20 quoted lines
> David Turner <dturner@twopensource.com> writes:
> 
> > I was looking at this again today, and noticed that it doesn't
> > really
> > address the HTTP case.
> > 
> > The central problem is that protocol v2 goes like this:
> > server: I have capabilities w,x,y, and z
> > client: I want capabilities x and z.
> > 
> > But HTTP goes like this:
> > client: [request]
> > server: [response]
> 
> I wonder if that can be solved by speculative request?
> 
> Let the connection initiator say "If you can do x and z, please do
> so", and allow the responder to say either "OK, I can do x and z; by
> the way the full capabilites I support are w, x, y and z", or
> "Sorry, can't do that; I have capabilities w, x, and y".

That protocol is somewhat more complicated. And it's different than the v2 protocol for the other transports (unless you are thinking that we should change those too?). It's actually a tiny bit like what I originally proposed for HTTP.

It sounds OK to me, but I want to know what others think before I start implementing.

Jeff King· May 25, 2016, 21:29 UTC · re: David Turner · lore

Re: [WIP PATCH 00/14] Protocol v2 patches

On Tue, May 24, 2016 at 06:46:48PM -0400, David Turner wrote:
Show 11 quoted lines
> I tried to make libcurl do the receive-before-sending thing, but it
> doesn't seem to be designed for it (even if you prime things by sending
> a "hello" from the client first).  My thought was to hook up
> CURLOPT_READFUNCTION and CURLOPT_WRITEFUNCTION, and have the read
> function return CURL_READFUNC_PAUSE and then have the write (=client
> receiving data ) function unpause the reader (= client sending data)
> once it gets the capabilities.  But apparently pausing only works with
> chunked encoding, which seems to cause Apache's mod_cgi to fail.
> 
> Maybe I'm missing something.  Has anyone else ever made something like
> this work?

I don't think it can work in the general case. HTTP is not full-duplex, and you have to send off the request and wait for the response. Even if you could convince the client and git-http-backend to do it, you're going to get foiled by proxies, web server implementations, and other middle-men.

Show 6 quoted lines
> Of course, I could always use CURLOPT_CONNECT_ONLY to write my own HTTP
> client, but that seems pretty unreasonable.
> 
> I also looked to see if libcurl had websockets support, since that's
> one kind of bidirectional conversation over HTTP, but it doesn't seem
> to.

I would love to see us move to a true bidirectional HTTP-based protocol. It would clear up all of the drawbacks that the current HTTP protocol has, and I think we could generally recommend it entirely over using git://. But like you, I haven't figured out an easy way to do it.

I hoped that maybe HTTP/2 would solve some of that if we waited long enough for it to be adopted, but it doesn't look like there's anything out of the box. It seems like the recommended solutions still involve websockets. I might be wrong, though; this is very much outside my area of expertise.

> Another choice is to make a separate /capabilities endpoint that gets
> hit before /info/refs.  This is a bit bad because:
> (a) it's another HTTP request
Right, this is the extra round-trip I mentioned in:
  http://thread.gmane.org/gmane.comp.version-control.git/291640/focus=291951

I think you could get rid of it by making protocol v2 a true "client speaks first" protocol, which aligns better with how HTTP works (but if we do that, it would be nice to do it for _all_ of the transports, so they stay closer to each other). But...

Show 9 quoted lines
> (b) it adds implicit state to the HTTP conversation.  If multiple git
> servers were behind a load balancer, you might end up getting server A
> for /capabilities and server B for /info/refs, and those servers might
> have different capabilities.  This is not impossible when testing a git
> server upgrade on one machine before rolling it out to a whole fleet. 
>  Maybe the rule for clients re capabilities is that they can request
> whatever capabilities they want, but the server is free to ignore that
> request and send whatever data it feels like.  That's not great, but it
> should work (I think).

I think this is already the case today. Every non-trivial git-over-http request requires at least two HTTP requests: one to receive the server fetch advertisement, and the second to actually do the work (and in the fetch case, the have/want negotiation in the second one may actually span several requests).

The capabilities from the server come in the first request, and then the client sends back its capabilities in the second one. So if you are hitting multiple incompatible servers, the server may not understand your request. Likewise, if an upload-pack request takes multiple hits, we send up the client capabilities in each request.

I don't think quietly ignoring unknown capabilities is a good idea. The results would range from confusing breakages (e.g., ignored multi-ack or no-done capabilities) to subtly wrong behavior (e.g., a server which ignores "atomic" and proceeds with a half-failed push anyway). Given the rarity of the situation, it's probably better for the server to barf with an appropriate error message. That sucks for the user, but it's probably better than the alternatives.

-Peff

← back to recent threads