threads / patch / 59619

patch, 8 partsfetch: introduce machine-parseable output

Subject: [PATCH 0/8] fetch: introduce machine-parseable output

## tl;dr

120 messages between Apr 19, 2023 and May 15, 2023. Diffs are folded; open one to read it.

replies: 119people: 7as markdown or json

Patrick Steinhardt· Apr 19, 2023, 12:31 UTC · lore
Hi,

this is the second part of my quest to introduce a machine-parseable output for git-fetch(1) after the initial refactorings that have been merged via e9dffbc7f1 (Merge branch 'ps/fetch-ref-update-reporting', 2023-04-06).

Parsing the output of fetches is mostly impossible. It prettifies reference names that are about to be updated, doesn't print the old and new object IDs the refs are being updated from and to, and prints all of that information in nice columns. In short, it is designed to be read by humans rather than machines.

This makes it hard to use in a script way though, e.g. to learn about which references actually have been updated or which have not been updated. This patch series intends to fix that by introducing a new machine-parseable interface:

```
$ git fetch --output-format=porcelain --no-progress
  fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
* 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
* 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
* 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
* 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
* 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
```
The series is structured as following:
    - Patches 1 and 2 improve test coverage for output formats.
    - Patch 3 fixes a bug with the current output format.
    - Patch 4 to 6 perform some preliminary refactorings.
    - Patch 7 introduces a new `--output-format=` option for
      git-fetch(1) that allows the user to configure the output more
      directly.
    - Patch 8 introduces the new "porcelain" output format.
Patrick
Patrick Steinhardt (8):
  fetch: split out tests for output format
  fetch: add a test to exercise invalid output formats
  fetch: fix missing from-reference when fetching HEAD:foo
  fetch: introduce `display_format` enum
  fetch: move display format parsing into main function
  fetch: move option related variables into main function
  fetch: introduce new `--output-format` option
  fetch: introduce machine-parseable "porcelain" output format
 Documentation/config/fetch.txt  |   4 +-
 Documentation/fetch-options.txt |   5 +
 Documentation/git-fetch.txt     |  17 +-
 builtin/fetch.c                 | 406 +++++++++++++++++++-------------
 t/t5510-fetch.sh                |  53 -----
 t/t5574-fetch-output.sh         | 209 ++++++++++++++++
 6 files changed, 475 insertions(+), 219 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
-- 
2.40.0
Patrick Steinhardt· Apr 19, 2023, 12:31 UTC · re: Patrick Steinhardt · lore

[PATCH 1/8] fetch: split out tests for output format

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Show changes to 2 files +63 −53

t/t5510-fetch.sh, t/t5574-fetch-output.sh

diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index dc44da9c79..4f289063ce 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
new file mode 100755
index 0000000000..f91b654d38
--- /dev/null
+++ b/t/t5574-fetch-output.sh
@@ -0,0 +1,63 @@
+#!/bin/sh
+
+test_description='git fetch output format'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+
+test_expect_success 'fetch aligned output' '
+	git clone . full-output &&
+	test_commit looooooooooooong-tag &&
+	(
+		cd full-output &&
+		git -c fetch.output=full fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main                 -> origin/main
+	looooooooooooong-tag -> looooooooooooong-tag
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'fetch compact output' '
+	git clone . compact &&
+	test_commit extraaa &&
+	(
+		cd compact &&
+		git -c fetch.output=compact fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main       -> origin/*
+	extraaa    -> *
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success '--no-show-forced-updates' '
+	mkdir forced-updates &&
+	(
+		cd forced-updates &&
+		git init &&
+		test_commit 1 &&
+		test_commit 2
+	) &&
+	git clone forced-updates forced-update-clone &&
+	git clone forced-updates no-forced-update-clone &&
+	git -C forced-updates reset --hard HEAD~1 &&
+	(
+		cd forced-update-clone &&
+		git fetch --show-forced-updates origin 2>output &&
+		test_i18ngrep "(forced update)" output
+	) &&
+	(
+		cd no-forced-update-clone &&
+		git fetch --no-show-forced-updates origin 2>output &&
+		test_i18ngrep ! "(forced update)" output
+	)
+'
+
+test_done
-- 
2.40.0
Patrick Steinhardt· Apr 19, 2023, 12:31 UTC · re: Patrick Steinhardt · lore

[PATCH 2/8] fetch: add a test to exercise invalid output formats

Add a testcase that exercises the logic when an invalid output format is passed via the `fetch.output` configuration.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5574-fetch-output.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
Show changes to t/t5574-fetch-output.sh +17 −0
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index f91b654d38..0e45c27007 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -7,6 +7,23 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_cmp expect actual &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
-- 
2.40.0
Patrick Steinhardt· Apr 19, 2023, 12:31 UTC · re: Patrick Steinhardt · lore

[PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

When displaying reference updates, we print a line that looks similar to the following:

```
 * branch               master          -> master
```

The "branch" bit changes depending on what kind of reference we're updating, while both of the right-hand references are computed by stripping well-known prefixes like "refs/heads/" or "refs/tags".

The logic is kind of intertwined though and not easy to follow: we precompute both the kind (e.g. "branch") and the what, which is the abbreviated remote reference name, in `store_updated_refs()` and then pass it down the call chain to `display_ref_update()`.

There is a set of different cases here:
    - When the remote reference name is "HEAD" we assume no kind and
      will thus instead print "[new ref]". We keep what at the empty
      string.
    - When the remote reference name has a well-known prefix then the
      kind would be "branch", "tag" or "remote-tracking branch". The
      what is the reference with the well-known prefix stripped and in
      fact matches the output that `prettify_refname()` would return.
    - Otherwise, we'll again assume no kind and keep the what set to the
      fully qualified reference name.

Now there is a bug with the first case here, where the remote reference name is "HEAD". As noted, "what" will be set to the empty string. And that seems to be intentional because we also use this information to update the FETCH_HEAD, and in case we're updating HEAD we seemingly don't want to append that to our FETCH_HEAD value.

But as mentioned, we also use this value to display reference updates. And while the call to `display_ref_update()` correctly figures out that we meant "HEAD" when `what` is empty, the call to `update_local_ref()` doesn't. `update_local_ref()` will then call `display_ref_update()` with the empty string and cause the following broken output:

```
$ git fetch --dry-run origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands work as expected:

```
$ git fetch --dry-run origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD
$ git fetch --dry-run origin master
From https://github.com/git/git
 * branch                  master     -> FETCH_HEAD
 * branch                  master     -> origin/master
```

Fix this bug by instead unconditionally passing the full reference name to `display_ref_update()` which learns to call `prettify_refname()` on it. This does fix the above bug and is otherwise functionally the same as `prettify_refname()` would only ever strip the well-known prefixes just as intended. So at the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c         | 37 +++++++++++++++++++------------------
 t/t5574-fetch-output.sh | 19 +++++++++++++++++++
 2 files changed, 38 insertions(+), 18 deletions(-)
Show changes to 2 files +38 −18

builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index c310d89878..7c64f0c562 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index 0e45c27007..55f0f05b6a 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -54,6 +54,25 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
-- 
2.40.0
Jacob Keller· Apr 26, 2023, 19:20 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

On 4/19/2023 5:31 AM, Patrick Steinhardt wrote:
Show 75 quoted lines
> When displaying reference updates, we print a line that looks similar to
> the following:
> 
> ```
>  * branch               master          -> master
> ```
> 
> The "branch" bit changes depending on what kind of reference we're
> updating, while both of the right-hand references are computed by
> stripping well-known prefixes like "refs/heads/" or "refs/tags".
> 
> The logic is kind of intertwined though and not easy to follow: we
> precompute both the kind (e.g. "branch") and the what, which is the
> abbreviated remote reference name, in `store_updated_refs()` and then
> pass it down the call chain to `display_ref_update()`.
> 
> There is a set of different cases here:
> 
>     - When the remote reference name is "HEAD" we assume no kind and
>       will thus instead print "[new ref]". We keep what at the empty
>       string.
> 
>     - When the remote reference name has a well-known prefix then the
>       kind would be "branch", "tag" or "remote-tracking branch". The
>       what is the reference with the well-known prefix stripped and in
>       fact matches the output that `prettify_refname()` would return.
> 
>     - Otherwise, we'll again assume no kind and keep the what set to the
>       fully qualified reference name.
> 
> Now there is a bug with the first case here, where the remote reference
> name is "HEAD". As noted, "what" will be set to the empty string. And
> that seems to be intentional because we also use this information to
> update the FETCH_HEAD, and in case we're updating HEAD we seemingly
> don't want to append that to our FETCH_HEAD value.
> 
> But as mentioned, we also use this value to display reference updates.
> And while the call to `display_ref_update()` correctly figures out that
> we meant "HEAD" when `what` is empty, the call to `update_local_ref()`
> doesn't. `update_local_ref()` will then call `display_ref_update()` with
> the empty string and cause the following broken output:
> 
> ```
> $ git fetch --dry-run origin HEAD:foo
> From https://github.com/git/git
>  * [new ref]                          -> foo
> ```
> 
> The HEAD string is clearly missing from the left-hand side of the arrow,
> which is further stressed by the point that the following commands work
> as expected:
> 
> ```
> $ git fetch --dry-run origin HEAD
> From https://github.com/git/git
>  * branch                  HEAD       -> FETCH_HEAD
> 
> $ git fetch --dry-run origin master
> From https://github.com/git/git
>  * branch                  master     -> FETCH_HEAD
>  * branch                  master     -> origin/master
> ```
> 
> Fix this bug by instead unconditionally passing the full reference name
> to `display_ref_update()` which learns to call `prettify_refname()` on
> it. This does fix the above bug and is otherwise functionally the same
> as `prettify_refname()` would only ever strip the well-known prefixes
> just as intended. So at the same time, this also simplifies the code a
> bit.
> 
> Note that this patch also changes formatting of the block that computes
> the "kind" and "what" variables. This is done on purpose so that it is
> part of the diff, hopefully making the change easier to comprehend.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>

The commit message here has a lot of context, but I found it a bit hard to parse through, especially relative to the actual fix in code.

One suggestion was to load the paragraphs a bit more with the actual problem being solved first, before beginning a lot of the context.

We also discussed the block of format changes and felt a bit mixed on whether to include it or not. It does match the coding style guidelines, but there is no actual functional change made to those lines in this series.

I think its a good improvement, and it does force some extra context into the diff which makes reading the resulting change easier.

Patrick Steinhardt· Apr 27, 2023, 10:58 UTC · re: Jacob Keller · lore

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

On Wed, Apr 26, 2023 at 12:20:15PM -0700, Jacob Keller wrote:
> On 4/19/2023 5:31 AM, Patrick Steinhardt wrote:
[snip]
Show 12 quoted lines
> The commit message here has a lot of context, but I found it a bit hard
> to parse through, especially relative to the actual fix in code.
> 
> One suggestion was to load the paragraphs a bit more with the actual
> problem being solved first, before beginning a lot of the context.
> 
> We also discussed the block of format changes and felt a bit mixed on
> whether to include it or not. It does match the coding style guidelines,
> but there is no actual functional change made to those lines in this series.
> 
> I think its a good improvement, and it does force some extra context
> into the diff which makes reading the resulting change easier.

I initially really struggled to explain to myself why my change actually works, and that shows in the commit message. I've reworded it a bit and took some of the suggested changes that Glen posted to hopefully make this clearer.

Patrick
Jacob Keller· Apr 26, 2023, 19:21 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

On 4/19/2023 5:31 AM, Patrick Steinhardt wrote:
Show 20 quoted lines
>  
> +test_expect_success 'fetch output with HEAD and --dry-run' '
> +	test_when_finished "rm -rf head" &&
> +	git clone . head &&
> +
> +	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
> +	cat >expect <<-EOF &&
> +	From $(test-tool path-utils real_path .)/.
> +	 * branch            HEAD       -> FETCH_HEAD
> +	EOF
> +	test_cmp expect actual &&
> +
> +	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
> +	cat >expect <<-EOF &&
> +	From $(test-tool path-utils real_path .)/.
> +	 * [new ref]         HEAD       -> foo
> +	EOF
> +	test_cmp expect actual
> +'
> +

The test mentions HEAD and --dry-run, but the bug seems to exist regardless of whether --dry-run is used. I understand the use of --dry-run for testing fetch output so that you can repeatably run git fetch and get the same results.

The tests here should probably also have a test that covers fetch without --dry-run though.

Thanks, Jake

Patrick Steinhardt· Apr 27, 2023, 10:58 UTC · re: Jacob Keller · lore

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

On Wed, Apr 26, 2023 at 12:21:40PM -0700, Jacob Keller wrote:
Show 31 quoted lines
> 
> 
> On 4/19/2023 5:31 AM, Patrick Steinhardt wrote:
> >  
> > +test_expect_success 'fetch output with HEAD and --dry-run' '
> > +	test_when_finished "rm -rf head" &&
> > +	git clone . head &&
> > +
> > +	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
> > +	cat >expect <<-EOF &&
> > +	From $(test-tool path-utils real_path .)/.
> > +	 * branch            HEAD       -> FETCH_HEAD
> > +	EOF
> > +	test_cmp expect actual &&
> > +
> > +	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
> > +	cat >expect <<-EOF &&
> > +	From $(test-tool path-utils real_path .)/.
> > +	 * [new ref]         HEAD       -> foo
> > +	EOF
> > +	test_cmp expect actual
> > +'
> > +
> 
> The test mentions HEAD and --dry-run, but the bug seems to exist
> regardless of whether --dry-run is used. I understand the use of
> --dry-run for testing fetch output so that you can repeatably run git
> fetch and get the same results.
> 
> The tests here should probably also have a test that covers fetch
> without --dry-run though.
Makes sense, will amend!
Patrick
Glen Choo· Apr 26, 2023, 19:25 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

Rearranging the lines slightly,
Patrick Steinhardt <ps@pks.im> writes:
Show 32 quoted lines
> When displaying reference updates, we print a line that looks similar to
> the following:
>
> ```
>  * branch               master          -> master
> ```
>
> The "branch" bit changes depending on what kind of reference we're
> updating, while both of the right-hand references are computed by
> stripping well-known prefixes like "refs/heads/" or "refs/tags".
>
> [...]
>                   we also use this value to display reference updates.
> And while the call to `display_ref_update()` correctly figures out that
> we meant "HEAD" when `what` is empty, the call to `update_local_ref()`
> doesn't. `update_local_ref()` will then call `display_ref_update()` with
> the empty string and cause the following broken output:
>
> ```
> $ git fetch --dry-run origin HEAD:foo
> From https://github.com/git/git
>  * [new ref]                          -> foo
> ```
>
> [...]
>
> Fix this bug by instead unconditionally passing the full reference name
> to `display_ref_update()` which learns to call `prettify_refname()` on
> it. This does fix the above bug and is otherwise functionally the same
> as `prettify_refname()` would only ever strip the well-known prefixes
> just as intended. So at the same time, this also simplifies the code a
> bit.

The bug fix is obviously good. I'm surprised we hadn't caught this sooner.

As a nitpicky comment, the commit message goes into a lot of detail, which makes it tricky to read on its own (though the level of detail makes it easy to match to the diff, making the diff quite easy to follow). I would have found this easier to read by summarizing the high-level mental model before diving into the background, e.g.

  store_updated_refs() parses the remote ref name to create a 'note' to
  write to FETCH_HEAD. This note is usually the prettified ref name, so
  it is used to diplay ref updates (display_ref_update()). But if the
  remote ref is HEAD, the note is the empty string [insert bug
  description]. Instead, use the note only as a note and have
  display_ref_update() prettify the ref name itself...
Show 21 quoted lines
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index c310d89878..7c64f0c562 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
>  	}
>  
>  	width = (summary_width + strlen(summary) - gettext_width(summary));
> +	remote = prettify_refname(remote);
> +	local = prettify_refname(local);
>  
>  	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
>  	if (!display_state->compact_format)
> -		print_remote_to_local(display_state, remote, prettify_refname(local));
> +		print_remote_to_local(display_state, remote, local);
>  	else
> -		print_compact(display_state, remote, prettify_refname(local));
> +		print_compact(display_state, remote, local);
>  	if (error)
>  		strbuf_addf(&display_state->buf, "  (%s)", error);
>  	strbuf_addch(&display_state->buf, '\n');

As expected, we now prettify the name isntead of trusting the 'note' that came in the parameter...

Show 75 quoted lines
> @@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
>  static int update_local_ref(struct ref *ref,
>  			    struct ref_transaction *transaction,
>  			    struct display_state *display_state,
> -			    const char *remote, const struct ref *remote_ref,
> +			    const struct ref *remote_ref,
>  			    int summary_width)
>  {
>  	struct commit *current = NULL, *updated;
> @@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
>  	if (oideq(&ref->old_oid, &ref->new_oid)) {
>  		if (verbosity > 0)
>  			display_ref_update(display_state, '=', _("[up to date]"), NULL,
> -					   remote, ref->name, summary_width);
> +					   remote_ref->name, ref->name, summary_width);
>  		return 0;
>  	}
>  
> @@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
>  		 */
>  		display_ref_update(display_state, '!', _("[rejected]"),
>  				   _("can't fetch into checked-out branch"),
> -				   remote, ref->name, summary_width);
> +				   remote_ref->name, ref->name, summary_width);
>  		return 1;
>  	}
>  
> @@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
>  			r = s_update_ref("updating tag", ref, transaction, 0);
>  			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
>  					   r ? _("unable to update local ref") : NULL,
> -					   remote, ref->name, summary_width);
> +					   remote_ref->name, ref->name, summary_width);
>  			return r;
>  		} else {
>  			display_ref_update(display_state, '!', _("[rejected]"),
>  					   _("would clobber existing tag"),
> -					   remote, ref->name, summary_width);
> +					   remote_ref->name, ref->name, summary_width);
>  			return 1;
>  		}
>  	}
> @@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
>  		r = s_update_ref(msg, ref, transaction, 0);
>  		display_ref_update(display_state, r ? '!' : '*', what,
>  				   r ? _("unable to update local ref") : NULL,
> -				   remote, ref->name, summary_width);
> +				   remote_ref->name, ref->name, summary_width);
>  		return r;
>  	}
>  
> @@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
>  		r = s_update_ref("fast-forward", ref, transaction, 1);
>  		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
>  				   r ? _("unable to update local ref") : NULL,
> -				   remote, ref->name, summary_width);
> +				   remote_ref->name, ref->name, summary_width);
>  		strbuf_release(&quickref);
>  		return r;
>  	} else if (force || ref->force) {
> @@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
>  		r = s_update_ref("forced-update", ref, transaction, 1);
>  		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
>  				   r ? _("unable to update local ref") : _("forced update"),
> -				   remote, ref->name, summary_width);
> +				   remote_ref->name, ref->name, summary_width);
>  		strbuf_release(&quickref);
>  		return r;
>  	} else {
>  		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
> -				   remote, ref->name, summary_width);
> +				   remote_ref->name, ref->name, summary_width);
>  		return 1;
>  	}
>  }
...
Show 18 quoted lines
> @@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
>  					  display_state->url_len);
>  
>  			if (ref) {
> -				rc |= update_local_ref(ref, transaction, display_state, what,
> +				rc |= update_local_ref(ref, transaction, display_state,
>  						       rm, summary_width);
>  				free(ref);
>  			} else if (write_fetch_head || dry_run) {
> @@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
>  				 */
>  				display_ref_update(display_state, '*',
>  						   *kind ? kind : "branch", NULL,
> -						   *what ? what : "HEAD",
> +						   rm->name,
>  						   "FETCH_HEAD", summary_width);
>  			}
>  		}
and we stop passing the 'note' as a parameter. Looks good.
Show 19 quoted lines
> @@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
>  			if (!strcmp(rm->name, "HEAD")) {
>  				kind = "";
>  				what = "";
> -			}
> -			else if (skip_prefix(rm->name, "refs/heads/", &what))
> +			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
>  				kind = "branch";
> -			else if (skip_prefix(rm->name, "refs/tags/", &what))
> +			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
>  				kind = "tag";
> -			else if (skip_prefix(rm->name, "refs/remotes/", &what))
> +			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
>  				kind = "remote-tracking branch";
> -			else {
> +			} else {
>  				kind = "";
>  				what = rm->name;
>  			}

I really appreciate that this makes the patch easier to read. I don't really appreciate this sort of churn, but it _is_ following CodingGuidelines:

	- When there are multiple arms to a conditional and some of them
	  require braces, enclose even a single line block in braces for
	  consistency. E.g.:
		if (foo) {
			doit();
		} else {
			one();
			two();
			three();
		}

(I initialy thought it wasn't. Thanks to other Review Club participants for pointing this out).

Show 9 quoted lines
> diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
> index 0e45c27007..55f0f05b6a 100755
> --- a/t/t5574-fetch-output.sh
> +++ b/t/t5574-fetch-output.sh
> @@ -54,6 +54,25 @@ test_expect_success 'fetch compact output' '
>  	test_cmp expect actual
>  '
>  
> +test_expect_success 'fetch output with HEAD and --dry-run' '

The commit message and diff didn't imply that this is a --dry-run only bug. I tested locally, and it seems to reproduce without --dry-run too, so I think we should drop "--dry-run" from this test name. In a later patch, you also add a test for porcelain output with --dry-run, but since this test seems designed for just this bug, I think we can drop the later test.

Patrick Steinhardt· Apr 27, 2023, 10:58 UTC · re: Glen Choo · lore

Re: [PATCH 3/8] fetch: fix missing from-reference when fetching HEAD:foo

On Wed, Apr 26, 2023 at 12:25:53PM -0700, Glen Choo wrote:
Show 54 quoted lines
> Rearranging the lines slightly,
> 
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > When displaying reference updates, we print a line that looks similar to
> > the following:
> >
> > ```
> >  * branch               master          -> master
> > ```
> >
> > The "branch" bit changes depending on what kind of reference we're
> > updating, while both of the right-hand references are computed by
> > stripping well-known prefixes like "refs/heads/" or "refs/tags".
> >
> > [...]
> >                   we also use this value to display reference updates.
> > And while the call to `display_ref_update()` correctly figures out that
> > we meant "HEAD" when `what` is empty, the call to `update_local_ref()`
> > doesn't. `update_local_ref()` will then call `display_ref_update()` with
> > the empty string and cause the following broken output:
> >
> > ```
> > $ git fetch --dry-run origin HEAD:foo
> > From https://github.com/git/git
> >  * [new ref]                          -> foo
> > ```
> >
> > [...]
> >
> > Fix this bug by instead unconditionally passing the full reference name
> > to `display_ref_update()` which learns to call `prettify_refname()` on
> > it. This does fix the above bug and is otherwise functionally the same
> > as `prettify_refname()` would only ever strip the well-known prefixes
> > just as intended. So at the same time, this also simplifies the code a
> > bit.
> 
> 
> The bug fix is obviously good. I'm surprised we hadn't caught this
> sooner.
> 
> As a nitpicky comment, the commit message goes into a lot of detail,
> which makes it tricky to read on its own (though the level of detail
> makes it easy to match to the diff, making the diff quite easy to
> follow). I would have found this easier to read by summarizing the
> high-level mental model before diving into the background, e.g.
> 
> 
>   store_updated_refs() parses the remote ref name to create a 'note' to
>   write to FETCH_HEAD. This note is usually the prettified ref name, so
>   it is used to diplay ref updates (display_ref_update()). But if the
>   remote ref is HEAD, the note is the empty string [insert bug
>   description]. Instead, use the note only as a note and have
>   display_ref_update() prettify the ref name itself...
I like that and will use a variant of this.
[snip]
Show 16 quoted lines
> > diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
> > index 0e45c27007..55f0f05b6a 100755
> > --- a/t/t5574-fetch-output.sh
> > +++ b/t/t5574-fetch-output.sh
> > @@ -54,6 +54,25 @@ test_expect_success 'fetch compact output' '
> >  	test_cmp expect actual
> >  '
> >  
> > +test_expect_success 'fetch output with HEAD and --dry-run' '
> 
> The commit message and diff didn't imply that this is a --dry-run only
> bug. I tested locally, and it seems to reproduce without --dry-run too,
> so I think we should drop "--dry-run" from this test name. In a later
> patch, you also add a test for porcelain output with --dry-run, but
> since this test seems designed for just this bug, I think we can drop
> the later test.
True, I'll amend the test.
Patrick
Patrick Steinhardt· Apr 19, 2023, 12:31 UTC · re: Patrick Steinhardt · lore

[PATCH 4/8] fetch: introduce `display_format` enum

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a `enum display_format` that is more readily extensible.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)
Show changes to builtin/fetch.c +64 −37
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 7c64f0c562..e03fcd1b2f 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -784,7 +790,6 @@ static int refcol_width(const struct ref *ref, int compact_format)
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display foramt %d", display_state->format);
 	}
 }
 
@@ -904,30 +920,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
-- 
2.40.0
Patrick Steinhardt· Apr 19, 2023, 12:31 UTC · re: Patrick Steinhardt · lore

[PATCH 5/8] fetch: move display format parsing into main function

Parsing the display format happens inside of `display_state_init()`. As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a output format though that is intended to be parseable by machines, for example inside of a script. In that case it becomes a bit awkward of an interface if you have to call git-fetch(1) with the `fetch.output` config key set. We're thus going to introduce a new `--output-format` switch for git-fetch(1) so that the output format can be configured more directly.

This means we'll have to hook parsing of the display format into the command line options parser. Having the code to determine the actual output format scattered across two different sites is hard to reason about though.

Refactor the code such that callers are expected to pass the display format that is to be used into `display_state_init()`. This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
Show changes to builtin/fetch.c +24 −17
diff --git a/builtin/fetch.c b/builtin/fetch.c
index e03fcd1b2f..bcc156a9ce 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -788,14 +788,13 @@ static int refcol_width(const struct ref *ref, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -812,15 +811,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT: {
@@ -1614,7 +1604,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1700,7 +1691,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2076,7 +2067,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2143,7 +2135,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2155,6 +2147,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	int i;
 	const char *bundle_uri;
+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
@@ -2181,6 +2174,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
+		const char *format = "full";
+
+		git_config_get_string_tmp("fetch.output", &format);
+		if (!strcasecmp(format, "full"))
+			display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(format, "compact"))
+			display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", format);
+	}
+
 	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
 		recurse_submodules = recurse_submodules_cli;
 
@@ -2309,7 +2315,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   display_format);
 	} else {
 		int max_children = max_jobs;
 
-- 
2.40.0
Patrick Steinhardt· Apr 19, 2023, 12:31 UTC · re: Patrick Steinhardt · lore

[PATCH 6/8] fetch: move option related variables into main function

The options of git-fetch(1) which we pass to `parse_options()` are declared globally in `builtin/fetch.c`. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into `cmd_fetch()`. Move variables that were previously forced to be declared globally and which are only used by `cmd_fetch()` into function-local scope.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 194 ++++++++++++++++++++++++------------------------
 1 file changed, 98 insertions(+), 96 deletions(-)
Show changes to builtin/fetch.c +98 −96
diff --git a/builtin/fetch.c b/builtin/fetch.c
index bcc156a9ce..81581b0033 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -79,9 +79,8 @@ static int all, append, dry_run, force, keep, multiple, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
@@ -160,92 +153,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2145,13 +2052,108 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
-	int i;
 	const char *bundle_uri;
+	const char *submodule_prefix = "";
 	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
-- 
2.40.0
Patrick Steinhardt· Apr 19, 2023, 12:31 UTC · re: Patrick Steinhardt · lore

[PATCH 7/8] fetch: introduce new `--output-format` option

It is only possible to configure the output format that git-fetch(1) uses by setting it via a config key. While this interface may be fine as long as we only have the current "full" and "compact" output formats, where it is unlikely that the user will have to change them regularly. But we're about to introduce a new machine-parseable interface where the current mechanism feels a little bit indirect and rigid.

Introduce a new `--output-format` option that allows the user to change the desired output format more directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/fetch-options.txt |  5 ++++
 builtin/fetch.c                 | 17 +++++++++++
 t/t5574-fetch-output.sh         | 50 +++++++++++++++++++++++++--------
 3 files changed, 60 insertions(+), 12 deletions(-)
Show changes to 3 files +60 −12

Documentation/fetch-options.txt, builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 622bd84768..654f96f79d 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -78,6 +78,11 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--output-format::
+	Control how ref update status is printed. Valid values are
+	`full` and `compact`. Default value is `full`. See section
+	OUTPUT in linkgit:git-fetch[1] for detail.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 81581b0033..22ba75a958 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -2050,6 +2050,21 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	return exit_code;
 }
 
+static int opt_parse_output_format(const struct option *opt, const char *arg, int unset)
+{
+	enum display_format *format = opt->value;
+	if (unset || !arg)
+		return 1;
+	else if (!strcmp(arg, "full"))
+		*format = DISPLAY_FORMAT_FULL;
+	else if (!strcmp(arg, "compact"))
+		*format = DISPLAY_FORMAT_COMPACT;
+	else
+		return error(_("unsupported output format '%s'"), arg);
+
+	return 0;
+}
+
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	const char *bundle_uri;
@@ -2101,6 +2116,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_CALLBACK(0, "output-format", &display_format, N_("format"), N_("output format"),
+			     opt_parse_output_format),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index 55f0f05b6a..5144d5ed21 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -24,14 +24,37 @@ test_expect_success 'fetch with invalid output format configuration' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch with invalid output format via command line' '
+	test_must_fail git fetch --output-format >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	error: option \`output-format${SQ} requires a value
+	EOF
+	test_cmp expect actual &&
+
+	test_must_fail git fetch --output-format= origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	error: unsupported output format ${SQ}${SQ}
+	EOF
+	test_cmp expect actual &&
+
+	test_must_fail git fetch --output-format=garbage origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	error: unsupported output format ${SQ}garbage${SQ}
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
+	test_when_finished "rm -rf full-cfg full-cli" &&
+	git clone . full-cfg &&
+	git clone . full-cli &&
 	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
+
+	git -C full-cfg -c fetch.output=full fetch origin >actual-cfg 2>&1 &&
+	git -C full-cli fetch --output-format=full origin >actual-cli 2>&1 &&
+	test_cmp actual-cfg actual-cli &&
+
+	grep -e "->" actual-cfg | cut -c 22- >actual &&
 	cat >expect <<-\EOF &&
 	main                 -> origin/main
 	looooooooooooong-tag -> looooooooooooong-tag
@@ -40,13 +63,16 @@ test_expect_success 'fetch aligned output' '
 '
 
 test_expect_success 'fetch compact output' '
-	git clone . compact &&
+	test_when_finished "rm -rf compact-cfg compact-cli" &&
+	git clone . compact-cli &&
+	git clone . compact-cfg &&
 	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
+
+	git -C compact-cfg -c fetch.output=compact fetch origin >actual-cfg 2>&1 &&
+	git -C compact-cli fetch --output-format=compact origin >actual-cli 2>&1 &&
+	test_cmp actual-cfg actual-cli &&
+
+	grep -e "->" actual-cfg | cut -c 22- >actual &&
 	cat >expect <<-\EOF &&
 	main       -> origin/*
 	extraaa    -> *
-- 
2.40.0
Glen Choo· Apr 26, 2023, 19:40 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 7/8] fetch: introduce new `--output-format` option

Patrick Steinhardt <ps@pks.im> writes:
Show 9 quoted lines
> @@ -2101,6 +2116,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
>  			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
>  		OPT_BOOL(0, "dry-run", &dry_run,
>  			 N_("dry run")),
> +		OPT_CALLBACK(0, "output-format", &display_format, N_("format"), N_("output format"),
> +			     opt_parse_output_format),
>  		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
>  			 N_("write fetched references to the FETCH_HEAD file")),
>  		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),

This change is good enough for fetching from a single remote, but if we want to support "--all", we'd also need to propagate the CLI flag to the child "fetch" processes. (The config option wouldn't have this bug because the child processes would parse config and get the correct value.)

Patrick Steinhardt· Apr 27, 2023, 10:58 UTC · re: Glen Choo · lore

Re: [PATCH 7/8] fetch: introduce new `--output-format` option

On Wed, Apr 26, 2023 at 12:40:13PM -0700, Glen Choo wrote:
Show 17 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > @@ -2101,6 +2116,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
> >  			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
> >  		OPT_BOOL(0, "dry-run", &dry_run,
> >  			 N_("dry run")),
> > +		OPT_CALLBACK(0, "output-format", &display_format, N_("format"), N_("output format"),
> > +			     opt_parse_output_format),
> >  		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
> >  			 N_("write fetched references to the FETCH_HEAD file")),
> >  		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
> 
> This change is good enough for fetching from a single remote, but if we
> want to support "--all", we'd also need to propagate the CLI flag to the
> child "fetch" processes. (The config option wouldn't have this bug
> because the child processes would parse config and get the correct
> value.)
Oh, right, good catch. Will fix, thanks!
Patrick
Patrick Steinhardt· Apr 19, 2023, 12:31 UTC · re: Patrick Steinhardt · lore

[PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

``` <flag> <old-object-id> <new-object-id> <local-reference>\n ```

We assume two conditions which are generally true:
    - The old and new object IDs have fixed known widths and cannot
      contain spaces.
    - References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/config/fetch.txt  |  4 +-
 Documentation/fetch-options.txt |  4 +-
 Documentation/git-fetch.txt     | 17 ++++++-
 builtin/fetch.c                 | 50 +++++++++++++++-----
 t/t5574-fetch-output.sh         | 84 +++++++++++++++++++++++++++++++++
 5 files changed, 142 insertions(+), 17 deletions(-)
Show changes to 5 files +142 −17

Documentation/config/fetch.txt, Documentation/fetch-options.txt, Documentation/git-fetch.txt, builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt
index 568f0f75b3..70734226c0 100644
--- a/Documentation/config/fetch.txt
+++ b/Documentation/config/fetch.txt
@@ -52,8 +52,8 @@ fetch.pruneTags::
 
 fetch.output::
 	Control how ref update status is printed. Valid values are
-	`full` and `compact`. Default value is `full`. See section
-	OUTPUT in linkgit:git-fetch[1] for detail.
+	`full`, `compact` and `porcelain`. Default value is `full`.
+	See section OUTPUT in linkgit:git-fetch[1] for detail.
 
 fetch.negotiationAlgorithm::
 	Control how information about the commits in the local repository
diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 654f96f79d..5ca8a67fe8 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -80,8 +80,8 @@ linkgit:git-config[1].
 
 --output-format::
 	Control how ref update status is printed. Valid values are
-	`full` and `compact`. Default value is `full`. See section
-	OUTPUT in linkgit:git-fetch[1] for detail.
+	`full`, `compact` and `porcelain`. Default value is `full`.
+	See section OUTPUT in linkgit:git-fetch[1] for detail.
 
 ifndef::git-pull[]
 --[no-]write-fetch-head::
diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt
index fba66f1460..efd22cd372 100644
--- a/Documentation/git-fetch.txt
+++ b/Documentation/git-fetch.txt
@@ -197,13 +197,26 @@ The output of "git fetch" depends on the transport method used; this
 section describes the output when fetching over the Git protocol
 (either locally or via ssh) and Smart HTTP protocol.
 
-The status of the fetch is output in tabular form, with each line
-representing the status of a single ref. Each line is of the form:
+The output format can be chosen either via the `fetch.output` config
+(see linkgit:git-config[1]), or via the `--output-format` switch.
+Supported values include:
+
+For the `full` and `compact` output formats, the status of the fetch is
+output in tabular, with each line representing the status of a single
+ref. Each line is of the form:
 
 -------------------------------
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+The `porcelain` output format is intended to be machine-parseable. In
+contrast to the human-readable output formats it thus prints to standard
+output instead of standard error. Each line is of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 22ba75a958..a99e24448b 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -52,6 +52,7 @@ enum display_format {
 	DISPLAY_FORMAT_UNKNOWN = 0,
 	DISPLAY_FORMAT_FULL,
 	DISPLAY_FORMAT_COMPACT,
+	DISPLAY_FORMAT_PORCELAIN,
 };
 
 struct display_state {
@@ -745,8 +746,11 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 		break;
 	}
+	case DISPLAY_FORMAT_PORCELAIN:
+		/* We don't need to precompute anything here. */
+		break;
 	default:
-		BUG("unexpected display foramt %d", display_state->format);
+		BUG("unexpected display format %d", display_state->format);
 	}
 }
 
@@ -815,8 +819,12 @@ static void print_compact(struct display_state *display_state,
 static void display_ref_update(struct display_state *display_state, char code,
 			       const char *summary, const char *error,
 			       const char *remote, const char *local,
+			       const struct object_id *old_oid,
+			       const struct object_id *new_oid,
 			       int summary_width)
 {
+	FILE *f = stderr;
+
 	if (verbosity < 0)
 		return;
 
@@ -849,12 +857,17 @@ static void display_ref_update(struct display_state *display_state, char code,
 
 		break;
 	}
+	case DISPLAY_FORMAT_PORCELAIN:
+		strbuf_addf(&display_state->buf, "%c %s %s %s", code,
+			    oid_to_hex(old_oid), oid_to_hex(new_oid), local);
+		f = stdout;
+		break;
 	default:
 		BUG("unexpected display format %d", display_state->format);
 	};
 	strbuf_addch(&display_state->buf, '\n');
 
-	fputs(display_state->buf.buf, stderr);
+	fputs(display_state->buf.buf, f);
 }
 
 static int update_local_ref(struct ref *ref,
@@ -872,7 +885,8 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 		return 0;
 	}
 
@@ -885,7 +899,8 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 
@@ -896,12 +911,14 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return 1;
 		}
 	}
@@ -934,7 +951,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return r;
 	}
 
@@ -956,7 +974,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -968,12 +987,14 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 }
@@ -1214,7 +1235,9 @@ static int store_updated_refs(struct display_state *display_state,
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
 						   rm->name,
-						   "FETCH_HEAD", summary_width);
+						   "FETCH_HEAD",
+						   &rm->new_oid, &rm->old_oid,
+						   summary_width);
 			}
 		}
 	}
@@ -1354,6 +1377,7 @@ static int prune_refs(struct display_state *display_state,
 		for (ref = stale_refs; ref; ref = ref->next) {
 			display_ref_update(display_state, '-', _("[deleted]"), NULL,
 					   _("(none)"), ref->name,
+					   &ref->new_oid, &ref->old_oid,
 					   summary_width);
 			warn_dangling_symref(stderr, dangling_msg, ref->name);
 		}
@@ -2059,6 +2083,8 @@ static int opt_parse_output_format(const struct option *opt, const char *arg, in
 		*format = DISPLAY_FORMAT_FULL;
 	else if (!strcmp(arg, "compact"))
 		*format = DISPLAY_FORMAT_COMPACT;
+	else if (!strcmp(arg, "porcelain"))
+		*format = DISPLAY_FORMAT_PORCELAIN;
 	else
 		return error(_("unsupported output format '%s'"), arg);
 
@@ -2201,6 +2227,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			display_format = DISPLAY_FORMAT_FULL;
 		else if (!strcasecmp(format, "compact"))
 			display_format = DISPLAY_FORMAT_COMPACT;
+		else if (!strcasecmp(format, "porcelain"))
+			display_format = DISPLAY_FORMAT_PORCELAIN;
 		else
 			die(_("invalid value for '%s': '%s'"),
 			    "fetch.output", format);
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index 5144d5ed21..28d02a93e4 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -80,6 +80,72 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch porcelain output' '
+	test_when_finished "rm -rf porcelain-cfg porcelain-cli" &&
+
+	# Set up a bunch of references that we can use to demonstrate different
+	# kinds of flag symbols in the output format.
+	MAIN_OLD=$(git rev-parse HEAD) &&
+	git branch "fast-forward" &&
+	git branch "deleted-branch" &&
+	git checkout -b force-updated &&
+	test_commit --no-tag force-update-old &&
+	FORCE_UPDATED_OLD=$(git rev-parse HEAD) &&
+	git checkout main &&
+
+	# Clone and pre-seed the repositories. We fetch references into two
+	# namespaces so that we can test that rejected and force-updated
+	# references are reported properly.
+	refspecs="refs/heads/*:refs/unforced/* +refs/heads/*:refs/forced/*" &&
+	git clone . porcelain-cli &&
+	git clone . porcelain-cfg &&
+	git -C porcelain-cfg fetch origin $refspecs &&
+	git -C porcelain-cli fetch origin $refspecs &&
+
+	# Now that we have set up the client repositories we can change our
+	# local references.
+	git branch new-branch &&
+	git branch -d deleted-branch &&
+	git checkout fast-forward &&
+	test_commit --no-tag fast-forward-new &&
+	FAST_FORWARD_NEW=$(git rev-parse HEAD) &&
+	git checkout force-updated &&
+	git reset --hard HEAD~ &&
+	test_commit --no-tag force-update-new &&
+	FORCE_UPDATED_NEW=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	- $MAIN_OLD $ZERO_OID refs/forced/deleted-branch
+	- $MAIN_OLD $ZERO_OID refs/unforced/deleted-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/unforced/fast-forward
+	! $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/unforced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/unforced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/forced/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/forced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/forced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/remotes/origin/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/remotes/origin/force-updated
+	* $ZERO_OID $MAIN_OLD refs/remotes/origin/new-branch
+	EOF
+
+	# Execute a dry-run fetch first. We do this to assert that the dry-run
+	# and non-dry-run fetches produces the same output. Execution of the
+	# fetch is expected to fail as we have a rejected reference update.
+	test_must_fail git -C porcelain-cfg -c fetch.output=porcelain fetch --dry-run --prune origin $refspecs >actual-dry-run-cfg &&
+	test_must_fail git -C porcelain-cli fetch --output-format=porcelain --dry-run --prune origin $refspecs >actual-dry-run-cli &&
+	test_cmp actual-dry-run-cfg actual-dry-run-cli &&
+	test_cmp expect actual-dry-run-cfg &&
+
+	# And now we perform a non-dry-run fetch.
+	test_must_fail git -C porcelain-cfg -c fetch.output=porcelain fetch --prune origin $refspecs >actual-cfg &&
+	test_must_fail git -C porcelain-cli fetch --output-format=porcelain --prune origin $refspecs >actual-cli &&
+	test_cmp actual-cfg actual-cli &&
+	test_cmp expect actual-cfg &&
+
+	# Ensure that the dry-run and non-dry-run output matches.
+	test_cmp actual-dry-run-cfg actual-cfg
+'
+
 test_expect_success 'fetch output with HEAD and --dry-run' '
 	test_when_finished "rm -rf head" &&
 	git clone . head &&
@@ -99,6 +165,24 @@ test_expect_success 'fetch output with HEAD and --dry-run' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+	COMMIT_ID=$(git rev-parse HEAD) &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID refs/heads/foo
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
-- 
2.40.0
Glen Choo· Apr 26, 2023, 19:52 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

Patrick Steinhardt <ps@pks.im> writes:
Show 5 quoted lines
> The output format is quite simple:
>
> ```
> <flag> <old-object-id> <new-object-id> <local-reference>\n
> ```

This format doesn't show the remote name or url that was fetched. That seems okay when fetching with a single remote, but it seems necessary with "--all". Perhaps you were planning to add that in a later series? If so, I think it's okay to call the "porcelain" format experimental, and forbid porcelain + --all until then.

Show 7 quoted lines
>
> We assume two conditions which are generally true:
>
>     - The old and new object IDs have fixed known widths and cannot
>       contain spaces.
>
>     - References cannot contain newlines.

This seems like a non-issue if we add a -z CLI option to indicate that entries should be NUL terminated instead of newline terminated, but that can be done as a followup.

Show 6 quoted lines
> With these assumptions, the output format becomes unambiguously
> parseable. Furthermore, given that this output is designed to be
> consumed by scripts, the machine-readable data is printed to stdout
> instead of stderr like the human-readable output is. This is mostly done
> so that other data printed to stderr, like error messages or progress
> meters, don't interfere with the parseable data.

Sending the 'main output' to stdout makes sense to me, but this (and possibly respecting -z) sounds like a different mode of operation, not just a matter of formats. It seems different enough that I'd prefer not to piggyback on "fetch.output" for this (even though this adds more surface to the interface...).

We could add --porcelain and say that "fetch.output" is ignored if --porcelain is also given. That also eliminates the need for --output-format, I think.

The .c changes look good to me.
Show 17 quoted lines
> +test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
> +	test_when_finished "rm -rf head" &&
> +	git clone . head &&
> +	COMMIT_ID=$(git rev-parse HEAD) &&
> +
> +	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
> +	cat >expect <<-EOF &&
> +	* $ZERO_OID $COMMIT_ID FETCH_HEAD
> +	EOF
> +	test_cmp expect actual &&
> +
> +	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
> +	cat >expect <<-EOF &&
> +	* $ZERO_OID $COMMIT_ID refs/heads/foo
> +	EOF
> +	test_cmp expect actual
> +'

As mentioned upthread, I think this test isn't needed because "porcelain" wouldn't run into the bug we are checking for anyway.

Patrick Steinhardt· Apr 27, 2023, 10:58 UTC · re: Glen Choo · lore

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

On Wed, Apr 26, 2023 at 12:52:46PM -0700, Glen Choo wrote:
Show 13 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > The output format is quite simple:
> >
> > ```
> > <flag> <old-object-id> <new-object-id> <local-reference>\n
> > ```
> 
> This format doesn't show the remote name or url that was fetched. That
> seems okay when fetching with a single remote, but it seems necessary
> with "--all". Perhaps you were planning to add that in a later series?
> If so, I think it's okay to call the "porcelain" format experimental,
> and forbid porcelain + --all until then.

The reason is mostly that I didn't find an output format that I really liked here. We'd basically have to repeat the remote URL for every single reference: just repeating it once per remote doesn't fly because with `--parallel` the output could be intermingled. But doing that feels wasteful to me, so I bailed. I guess I'm also biased here because it just wouldn't be useful to myself.

So with that in mind, I'd like to continue ignoring this issue for now and just not report the remote that the ref came from. But I'd also argue that we don't have to restrict porcelain mode to single-remote fetches: it can still be useful to do multi-remote fetches even without the information where a certain reference update comes from. So any kind of restriction would feel artificial to me here.

Furthermore, I'd argue that it is not necessary to label the format as experimental only because of this limitation. With the refactorings done in this and the preceding patch series it is easy to add a new format in case there indeed is somebody that would have a usecase for this. The "porcelain" format should stay stable, and if we decide that we want to also report the remote for each reference in a follow-up we can easily add a "porcelain-v2" or "porcelain-with-remote" format.

As I said though: I'm clearly biased, so if you feel like my train of though is simply me being lazy then I'd carve in and adapt.

Show 10 quoted lines
> > We assume two conditions which are generally true:
> >
> >     - The old and new object IDs have fixed known widths and cannot
> >       contain spaces.
> >
> >     - References cannot contain newlines.
> 
> This seems like a non-issue if we add a -z CLI option to indicate that
> entries should be NUL terminated instead of newline terminated, but that
> can be done as a followup.

Yeah, either via `-z` or a new porcelain output format. But both of these conditions should generally be true anyway, so I don't see that those should become a problem.

Show 16 quoted lines
> > With these assumptions, the output format becomes unambiguously
> > parseable. Furthermore, given that this output is designed to be
> > consumed by scripts, the machine-readable data is printed to stdout
> > instead of stderr like the human-readable output is. This is mostly done
> > so that other data printed to stderr, like error messages or progress
> > meters, don't interfere with the parseable data.
> 
> Sending the 'main output' to stdout makes sense to me, but this (and
> possibly respecting -z) sounds like a different mode of operation, not
> just a matter of formats. It seems different enough that I'd prefer not
> to piggyback on "fetch.output" for this (even though this adds more
> surface to the interface...).
> 
> We could add --porcelain and say that "fetch.output" is ignored if
> --porcelain is also given. That also eliminates the need for
> --output-format, I think.

I was thinking about this initially, as well. But ultimately I decided against this especially because of your second paragraph: we'd now need to think about precedence of options and mutual exclusion, and that to me feels like an interface that is less obvious than a single knob that works as you'd expect.

Show 22 quoted lines
> The .c changes look good to me.
> 
> > +test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
> > +	test_when_finished "rm -rf head" &&
> > +	git clone . head &&
> > +	COMMIT_ID=$(git rev-parse HEAD) &&
> > +
> > +	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
> > +	cat >expect <<-EOF &&
> > +	* $ZERO_OID $COMMIT_ID FETCH_HEAD
> > +	EOF
> > +	test_cmp expect actual &&
> > +
> > +	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
> > +	cat >expect <<-EOF &&
> > +	* $ZERO_OID $COMMIT_ID refs/heads/foo
> > +	EOF
> > +	test_cmp expect actual
> > +'
> 
> As mentioned upthread, I think this test isn't needed because
> "porcelain" wouldn't run into the bug we are checking for anyway.

The only reason that the other bug was able to survive for so long was that we didn't have test coverage there. So I think it makes sense to explicitly test this, too, also because it causes us to walk a different code path.

Last but not least: this test uncovered a segfault I had in a previous version. So I'd rather keep it :)

Patrick
Glen Choo· Apr 27, 2023, 23:20 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

Patrick Steinhardt <ps@pks.im> writes:
Show 21 quoted lines
> On Wed, Apr 26, 2023 at 12:52:46PM -0700, Glen Choo wrote:
>> Patrick Steinhardt <ps@pks.im> writes:
>> 
>> > The output format is quite simple:
>> >
>> > ```
>> > <flag> <old-object-id> <new-object-id> <local-reference>\n
>> > ```
>> 
>> This format doesn't show the remote name or url that was fetched. That
>> seems okay when fetching with a single remote, but it seems necessary
>> with "--all". Perhaps you were planning to add that in a later series?
>> If so, I think it's okay to call the "porcelain" format experimental,
>> and forbid porcelain + --all until then.
>
> The reason is mostly that I didn't find an output format that I really
> liked here. We'd basically have to repeat the remote URL for every
> single reference: just repeating it once per remote doesn't fly because
> with `--parallel` the output could be intermingled. But doing that feels
> wasteful to me, so I bailed. I guess I'm also biased here because it
> just wouldn't be useful to myself.
Yeah, I couldn't think of a good way to output the remote URL either.
Show 14 quoted lines
> So with that in mind, I'd like to continue ignoring this issue for now
> and just not report the remote that the ref came from. But I'd also
> argue that we don't have to restrict porcelain mode to single-remote
> fetches: it can still be useful to do multi-remote fetches even without
> the information where a certain reference update comes from. So any kind
> of restriction would feel artificial to me here.
>
> Furthermore, I'd argue that it is not necessary to label the format as
> experimental only because of this limitation. With the refactorings done
> in this and the preceding patch series it is easy to add a new format in
> case there indeed is somebody that would have a usecase for this. The
> "porcelain" format should stay stable, and if we decide that we want to
> also report the remote for each reference in a follow-up we can easily
> add a "porcelain-v2" or "porcelain-with-remote" format.

I agree that the current form is useful, but it seems quite likely that users will want the remote name/url at some point. In the majority of cases, the user can parse the remote name from the updated local ref and if they want, they look up the url by asking "git config", but that breaks down quite quickly if they have a custom refspec.

My suggestion to label it experimental was meant to give us the freedom to change the "first version" in backwards-incompatible ways that we otherwise couldn't. We could say "here's porcelain-v2, use this and ignore porcelain-v1", but it would be even nicer if the user didn't need to think about porcelain-v1. The idea is that we could call it experimental, see what users need in the wild, then make changes without having to create a new "porcelain-v2" mode.

(I realize in hindsight that calling it experimental sounds a bit insulting. I thought your proposal was quite sensible, actually. i apologize if it came across that way)

Perhaps part of the reason why you'd prefer not to make it experimental is that users who might want this feature are too afraid of backwards-incompatible changes. I think that's fair. A different way of achieving the same flexibility would be to make the "porcelain" format extensible (like git status --porcelain=v2). A future, multivalued "--porcelain-options=show-remote-url,show-remote-ref" would cover our bases.

I'm happy with either approach as long as we don't have to bikeshed about the "perfect" porcelain output :)

Show 22 quoted lines
>> > With these assumptions, the output format becomes unambiguously
>> > parseable. Furthermore, given that this output is designed to be
>> > consumed by scripts, the machine-readable data is printed to stdout
>> > instead of stderr like the human-readable output is. This is mostly done
>> > so that other data printed to stderr, like error messages or progress
>> > meters, don't interfere with the parseable data.
>> 
>> Sending the 'main output' to stdout makes sense to me, but this (and
>> possibly respecting -z) sounds like a different mode of operation, not
>> just a matter of formats. It seems different enough that I'd prefer not
>> to piggyback on "fetch.output" for this (even though this adds more
>> surface to the interface...).
>> 
>> We could add --porcelain and say that "fetch.output" is ignored if
>> --porcelain is also given. That also eliminates the need for
>> --output-format, I think.
>
> I was thinking about this initially, as well. But ultimately I decided
> against this especially because of your second paragraph: we'd now need
> to think about precedence of options and mutual exclusion, and that to
> me feels like an interface that is less obvious than a single knob that
> works as you'd expect.
My full thoughts on this are in
  https://lore.kernel.org/git/kl6lildhlz3i.fsf@chooglen-macbookpro.roam.corp.google.com

but the short version is that I'm not sure if I expect something as innocuous-looking as --output-format would imply other, machine-friendly things (like stdout instead of stderr), and using --porcelain might make option precedence clearer in some situtations (like if -z is given).

Show 28 quoted lines
>> > +test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
>> > +	test_when_finished "rm -rf head" &&
>> > +	git clone . head &&
>> > +	COMMIT_ID=$(git rev-parse HEAD) &&
>> > +
>> > +	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
>> > +	cat >expect <<-EOF &&
>> > +	* $ZERO_OID $COMMIT_ID FETCH_HEAD
>> > +	EOF
>> > +	test_cmp expect actual &&
>> > +
>> > +	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
>> > +	cat >expect <<-EOF &&
>> > +	* $ZERO_OID $COMMIT_ID refs/heads/foo
>> > +	EOF
>> > +	test_cmp expect actual
>> > +'
>> 
>> As mentioned upthread, I think this test isn't needed because
>> "porcelain" wouldn't run into the bug we are checking for anyway.
>
> The only reason that the other bug was able to survive for so long was
> that we didn't have test coverage there. So I think it makes sense to
> explicitly test this, too, also because it causes us to walk a different
> code path.
>
> Last but not least: this test uncovered a segfault I had in a previous
> version. So I'd rather keep it :)

Ah, okay, both are fair. In that case, it probably makes sense to drop the "--dry-run".

Patrick Steinhardt· Apr 28, 2023, 08:51 UTC · re: Glen Choo · lore

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

On Thu, Apr 27, 2023 at 04:20:12PM -0700, Glen Choo wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > On Wed, Apr 26, 2023 at 12:52:46PM -0700, Glen Choo wrote:
> >> Patrick Steinhardt <ps@pks.im> writes:
[snip]
Show 32 quoted lines
> > So with that in mind, I'd like to continue ignoring this issue for now
> > and just not report the remote that the ref came from. But I'd also
> > argue that we don't have to restrict porcelain mode to single-remote
> > fetches: it can still be useful to do multi-remote fetches even without
> > the information where a certain reference update comes from. So any kind
> > of restriction would feel artificial to me here.
> >
> > Furthermore, I'd argue that it is not necessary to label the format as
> > experimental only because of this limitation. With the refactorings done
> > in this and the preceding patch series it is easy to add a new format in
> > case there indeed is somebody that would have a usecase for this. The
> > "porcelain" format should stay stable, and if we decide that we want to
> > also report the remote for each reference in a follow-up we can easily
> > add a "porcelain-v2" or "porcelain-with-remote" format.
> 
> I agree that the current form is useful, but it seems quite likely that
> users will want the remote name/url at some point. In the majority of
> cases, the user can parse the remote name from the updated local ref and
> if they want, they look up the url by asking "git config", but that
> breaks down quite quickly if they have a custom refspec.
> 
> My suggestion to label it experimental was meant to give us the freedom
> to change the "first version" in backwards-incompatible ways that we
> otherwise couldn't. We could say "here's porcelain-v2, use this and
> ignore porcelain-v1", but it would be even nicer if the user didn't need
> to think about porcelain-v1. The idea is that we could call it
> experimental, see what users need in the wild, then make changes without
> having to create a new "porcelain-v2" mode.
> 
> (I realize in hindsight that calling it experimental sounds a bit
> insulting. I thought your proposal was quite sensible, actually. i
> apologize if it came across that way)
No offense taken, so no need to apologize.
> Perhaps part of the reason why you'd prefer not to make it experimental
> is that users who might want this feature are too afraid of
> backwards-incompatible changes. I think that's fair.

Exactly. I'm implementing this whole feature with a concrete usecase at GitLab in mind, and if we declare this as experimental then it will become a whole lot more risky for us to actually adopt it. Because if the output format may change at any point in time, then the code that builds on top of it is likely to be broken.

Show 5 quoted lines
> A different way of
> achieving the same flexibility would be to make the "porcelain" format
> extensible (like git status --porcelain=v2). A future, multivalued
> "--porcelain-options=show-remote-url,show-remote-ref" would cover
> our bases.

Yeah, that would indeed be quite flexible. Theoretically, there is no reason why we couldn't have `--output-format=porcelain,show-remote-url` though.

> I'm happy with either approach as long as we don't have to bikeshed
> about the "perfect" porcelain output :)

Agreed, and that's why I'm currently defending the "good enough" format. It should likely work for most usecases that exist out there. The target audience is going to be quite small here as this is not a user-directed feature. Furthermore, I assume that the overlap of machines that want to parse all reference updates done in a multi-remote fetch while also being able to exactly tell which remote brought in what updates is going to be tiny.

Show 31 quoted lines
> >> > With these assumptions, the output format becomes unambiguously
> >> > parseable. Furthermore, given that this output is designed to be
> >> > consumed by scripts, the machine-readable data is printed to stdout
> >> > instead of stderr like the human-readable output is. This is mostly done
> >> > so that other data printed to stderr, like error messages or progress
> >> > meters, don't interfere with the parseable data.
> >> 
> >> Sending the 'main output' to stdout makes sense to me, but this (and
> >> possibly respecting -z) sounds like a different mode of operation, not
> >> just a matter of formats. It seems different enough that I'd prefer not
> >> to piggyback on "fetch.output" for this (even though this adds more
> >> surface to the interface...).
> >> 
> >> We could add --porcelain and say that "fetch.output" is ignored if
> >> --porcelain is also given. That also eliminates the need for
> >> --output-format, I think.
> >
> > I was thinking about this initially, as well. But ultimately I decided
> > against this especially because of your second paragraph: we'd now need
> > to think about precedence of options and mutual exclusion, and that to
> > me feels like an interface that is less obvious than a single knob that
> > works as you'd expect.
> 
> My full thoughts on this are in
> 
>   https://lore.kernel.org/git/kl6lildhlz3i.fsf@chooglen-macbookpro.roam.corp.google.com
> 
> but the short version is that I'm not sure if I expect something as
> innocuous-looking as --output-format would imply other, machine-friendly
> things (like stdout instead of stderr), and using --porcelain might make
> option precedence clearer in some situtations (like if -z is given).

I'm not even sure that `-z` makes sense in this context. If we see cases where not using `-z` can cause the machine-readable interface to become unparseable then this is a bug in the output format, if you ask me. Mostly because the whole intent of it is to be machine-parseable. So if we output data that can e.g. contain newlines, then we must not use newlines as part of the output format or alternatively escape them. Why let the author of the script shoot themselves into the foot?

Anyway, I'm digressing. It's hard for me to decide what to do right now. The thread with Junio and Jacob points into the direction of keeping the `--output-format=` interface, while this thread points into the other direction. I'm naturally more inclined to keep `--output-format=`, mostly because I personally feel like it's the more obvious interface. But I also see your point, so it's not really a choice of right-or-wrong here, but rather of style.

Patrick
Glen Choo· Apr 28, 2023, 17:20 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

Patrick Steinhardt <ps@pks.im> writes:
Show 7 quoted lines
>> I'm happy with either approach as long as we don't have to bikeshed
>> about the "perfect" porcelain output :)
>
> Agreed, and that's why I'm currently defending the "good enough" format.
> It should likely work for most usecases that exist out there. The target
> audience is going to be quite small here as this is not a user-directed
> feature.

Okay, I agree that this is "good enough" for most cases and we can extend it if needed.

Show 16 quoted lines
>> My full thoughts on this are in
>> 
>>   https://lore.kernel.org/git/kl6lildhlz3i.fsf@chooglen-macbookpro.roam.corp.google.com
>> 
>> but the short version is that I'm not sure if I expect something as
>> innocuous-looking as --output-format would imply other, machine-friendly
>> things (like stdout instead of stderr), and using --porcelain might make
>> option precedence clearer in some situtations (like if -z is given).
>
> I'm not even sure that `-z` makes sense in this context. If we see cases
> where not using `-z` can cause the machine-readable interface to become
> unparseable then this is a bug in the output format, if you ask me.
> Mostly because the whole intent of it is to be machine-parseable. So if
> we output data that can e.g. contain newlines, then we must not use
> newlines as part of the output format or alternatively escape them. Why
> let the author of the script shoot themselves into the foot?

Agreed. I think the current output format is resilient enough, just that "-z" is something we choose to support for plumbing anyway.

Show 7 quoted lines
> Anyway, I'm digressing. It's hard for me to decide what to do right now.
> The thread with Junio and Jacob points into the direction of keeping the
> `--output-format=` interface, while this thread points into the other
> direction. I'm naturally more inclined to keep `--output-format=`,
> mostly because I personally feel like it's the more obvious interface.
> But I also see your point, so it's not really a choice of right-or-wrong
> here, but rather of style.

Yes. All sides seem to understand the tradeoffs, but we value them differently.

I'd personally prefer to err on the side of consistency, because even if the existing behavior is less-than-ideal since:

1) This makes it possible for us to fix it in a consistent, well-thought
   out way, so we don't have to decide the future for the whole project
   in a git-fetch series.
2) At least it still behaves how _some_ users have come to expect it.

As for which is more consistent, a grep for --porcelain over Documentation/* shows:

- git-push
- git-status
- git-worktree
- git-blame
- git-commit

Most of them already output to stdout. The lone exception is git-push, which does exactly this "use stdout for the main output, but keep using stderr for debugging output" behavior you added, so using --porcelain seems somewhat consistent.

On the other hand, I couldn't find any other option that switches from stderr to stdout (we sometimes say --stdout to say "use stdout instead of a file", but that's different), so if we added this behavior to "git fetch --[output-]format", this might be the first.

(Sidenote: most instances of --format are for templated-style format, but "git-replace" does accept an enum for it. I don't think this should weigh heavily in our decision though.)

For the other, non git-push commands, --porcelain is treated like an output format and respects "last one wins" regular option precedence, e.g. in the case of "git status --porcelain --short". I find this behavior somewhat confusing (I'd expect --porcelain to win, or at least be incompatible), but at least we can consistently change this in the future [*].

That said, I'm willing to accept that I'm biased toward my own ideas. If others don't find "--format=porcelain implies stdout" confusing, I can accept that decision.

[*] In all likelihood, we probably can't make the change to --porcelain for backwards-compatibility reasons, but we could introduce a synonym with the behavior we want ("--machine"?) and have OPT_PARSE_MACHINE handle both --porcelain and --machine.

Felipe Contreras· May 2, 2023, 20:55 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 8/8] fetch: introduce machine-parseable "porcelain" output format

Patrick Steinhardt wrote:
Show 21 quoted lines
> On Wed, Apr 26, 2023 at 12:52:46PM -0700, Glen Choo wrote:
> > Patrick Steinhardt <ps@pks.im> writes:
> > 
> > > The output format is quite simple:
> > >
> > > ```
> > > <flag> <old-object-id> <new-object-id> <local-reference>\n
> > > ```
> > 
> > This format doesn't show the remote name or url that was fetched. That
> > seems okay when fetching with a single remote, but it seems necessary
> > with "--all". Perhaps you were planning to add that in a later series?
> > If so, I think it's okay to call the "porcelain" format experimental,
> > and forbid porcelain + --all until then.
> 
> The reason is mostly that I didn't find an output format that I really
> liked here. We'd basically have to repeat the remote URL for every
> single reference: just repeating it once per remote doesn't fly because
> with `--parallel` the output could be intermingled. But doing that feels
> wasteful to me, so I bailed. I guess I'm also biased here because it
> just wouldn't be useful to myself.

Couldn't each URL be assigned an index and then just show `0` in the output? I think it should be straightforward for a parser to figure out what `0` means but if not, that's something that could be figured out once the use case arrives (which very well could be never).

-- 
Felipe Contreras
Felipe Contreras· Apr 24, 2023, 20:17 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

Patrick Steinhardt wrote:
Show 20 quoted lines
> Parsing the output of fetches is mostly impossible. It prettifies
> reference names that are about to be updated, doesn't print the old and
> new object IDs the refs are being updated from and to, and prints all of
> that information in nice columns. In short, it is designed to be read by
> humans rather than machines.
> 
> This makes it hard to use in a script way though, e.g. to learn about
> which references actually have been updated or which have not been
> updated. This patch series intends to fix that by introducing a new
> machine-parseable interface:
> 
> ```
> $ git fetch --output-format=porcelain --no-progress
>   fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
> * 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
> * 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
> * 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
> * 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
> * 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
> ```

Makes sense, my only question is what other format could `git fetch` have? I think `--format=porcelain` is clear enough.

-- 
Felipe Contreras
Patrick Steinhardt· Apr 25, 2023, 09:58 UTC · re: Felipe Contreras · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

On Mon, Apr 24, 2023 at 02:17:31PM -0600, Felipe Contreras wrote:
Show 24 quoted lines
> Patrick Steinhardt wrote:
> > Parsing the output of fetches is mostly impossible. It prettifies
> > reference names that are about to be updated, doesn't print the old and
> > new object IDs the refs are being updated from and to, and prints all of
> > that information in nice columns. In short, it is designed to be read by
> > humans rather than machines.
> > 
> > This makes it hard to use in a script way though, e.g. to learn about
> > which references actually have been updated or which have not been
> > updated. This patch series intends to fix that by introducing a new
> > machine-parseable interface:
> > 
> > ```
> > $ git fetch --output-format=porcelain --no-progress
> >   fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
> > * 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
> > * 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
> > * 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
> > * 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
> > * 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
> > ```
> 
> Makes sense, my only question is what other format could `git fetch` have? I
> think `--format=porcelain` is clear enough.

Yeah, I'd be perfectly happy to rename this to `--format=porcelain`. I'll wait for the Review Club that discusses this patch set tomorrow and will send a new version with that change afterwards if nobody disagrees.

Patrick
Jacob Keller· Apr 26, 2023, 19:14 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

On 4/25/2023 2:58 AM, Patrick Steinhardt wrote:
Show 31 quoted lines
> On Mon, Apr 24, 2023 at 02:17:31PM -0600, Felipe Contreras wrote:
>> Patrick Steinhardt wrote:
>>> Parsing the output of fetches is mostly impossible. It prettifies
>>> reference names that are about to be updated, doesn't print the old and
>>> new object IDs the refs are being updated from and to, and prints all of
>>> that information in nice columns. In short, it is designed to be read by
>>> humans rather than machines.
>>>
>>> This makes it hard to use in a script way though, e.g. to learn about
>>> which references actually have been updated or which have not been
>>> updated. This patch series intends to fix that by introducing a new
>>> machine-parseable interface:
>>>
>>> ```
>>> $ git fetch --output-format=porcelain --no-progress
>>>   fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
>>> * 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
>>> * 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
>>> * 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
>>> * 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
>>> * 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
>>> ```
>>
>> Makes sense, my only question is what other format could `git fetch` have? I
>> think `--format=porcelain` is clear enough.
> 
> Yeah, I'd be perfectly happy to rename this to `--format=porcelain`.
> I'll wait for the Review Club that discusses this patch set tomorrow and
> will send a new version with that change afterwards if nobody disagrees.
> 
> Patrick

We had some discussion during review club about this, where the idea of using "--porcelain" came up because many commands use that when switching into a machine readable format.

In addition, this format not only changes the output but also moves it from being on stderr to stdout, which is a hint that the intended usage of the command is now a little different.

I don't have strong opinion here but want to note that "--format" is often used by commands like log which changes how we structure the output of git objects.

Obviously using "--porcelain" is a bit weird when dealing with the pre-existing compact and full outputs, and perhaps --format wouldn't be confusing.

I didn't find any command which used --output-format today, and all the uses of --format I saw were for object formatting like git log.

I'm ok with --format, but wanted to note the potential confusion.

git status also uses --porcelain and has a -z option for NUL terminating instead of newline terminating. We thought -z might be useful to handle the potential of weird refnames that include characters. No one on the review could remember what rules were enforced on refnames to confirm if it was legal to have '\n' in a refname or not.

Junio C Hamano· Apr 26, 2023, 20:23 UTC · re: Jacob Keller · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

Jacob Keller <jacob.e.keller@intel.com> writes:
Show 13 quoted lines
>> Yeah, I'd be perfectly happy to rename this to `--format=porcelain`.
>> I'll wait for the Review Club that discusses this patch set tomorrow and
>> will send a new version with that change afterwards if nobody disagrees.
>> 
>> Patrick
>
> We had some discussion during review club about this, where the idea of
> using "--porcelain" came up because many commands use that when
> switching into a machine readable format.
>
> In addition, this format not only changes the output but also moves it
> from being on stderr to stdout, which is a hint that the intended usage
> of the command is now a little different.

A little different from what? I do not think the answer would be "other program's --porcelain mode", as sending them to stdout would be one of the things that make the output easier for programs to parse, so it does sound like very much in the same spirit as "git status --porcelain" where its output format gets tweaked to be more machine friendly.

The output with "--porcelain" option enabled tend to be less human friendly and the distinction between Porcelain (for humans) and plumbing (for scripts) is reversed in the use of the word there---it started as "this is the option for those who write Porcelain commands to use", but still it is not a very good name for the option.

I am perfectly OK if the plan is to uniformly use --output-format (or something equally more descriptive) and migrate and deprecate the "--porcelain" option away from existing commands.

Thanks.
Jacob Keller· Apr 26, 2023, 20:30 UTC · re: Junio C Hamano · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

On 4/26/2023 1:23 PM, Junio C Hamano wrote:
Show 22 quoted lines
> Jacob Keller <jacob.e.keller@intel.com> writes:
> 
>>> Yeah, I'd be perfectly happy to rename this to `--format=porcelain`.
>>> I'll wait for the Review Club that discusses this patch set tomorrow and
>>> will send a new version with that change afterwards if nobody disagrees.
>>>
>>> Patrick
>>
>> We had some discussion during review club about this, where the idea of
>> using "--porcelain" came up because many commands use that when
>> switching into a machine readable format.
>>
>> In addition, this format not only changes the output but also moves it
>> from being on stderr to stdout, which is a hint that the intended usage
>> of the command is now a little different.
> 
> A little different from what?  I do not think the answer would be
> "other program's --porcelain mode", as sending them to stdout would
> be one of the things that make the output easier for programs to
> parse, so it does sound like very much in the same spirit as "git
> status --porcelain" where its output format gets tweaked to be more
> machine friendly.

A little different from using git fetch normally where all output is stderr and is generally "this is what I did" but which was obviously not intended to be parsed by scripts but instead by the human who ran the command.

Show 7 quoted lines
> 
> The output with "--porcelain" option enabled tend to be less human
> friendly and the distinction between Porcelain (for humans) and
> plumbing (for scripts) is reversed in the use of the word there---it
> started as "this is the option for those who write Porcelain
> commands to use", but still it is not a very good name for the
> option.

Yea the option sort of means "to be used by those implementing porcelain" and its definitely a bit confusing.

Show 6 quoted lines
> 
> I am perfectly OK if the plan is to uniformly use --output-format
> (or something equally more descriptive) and migrate and deprecate
> the "--porcelain" option away from existing commands.
> 
> Thanks.
That makes sense to me as well.
Patrick Steinhardt· Apr 27, 2023, 10:58 UTC · re: Junio C Hamano · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

On Wed, Apr 26, 2023 at 01:23:12PM -0700, Junio C Hamano wrote:
Show 29 quoted lines
> Jacob Keller <jacob.e.keller@intel.com> writes:
> 
> >> Yeah, I'd be perfectly happy to rename this to `--format=porcelain`.
> >> I'll wait for the Review Club that discusses this patch set tomorrow and
> >> will send a new version with that change afterwards if nobody disagrees.
> >> 
> >> Patrick
> >
> > We had some discussion during review club about this, where the idea of
> > using "--porcelain" came up because many commands use that when
> > switching into a machine readable format.
> >
> > In addition, this format not only changes the output but also moves it
> > from being on stderr to stdout, which is a hint that the intended usage
> > of the command is now a little different.
> 
> A little different from what?  I do not think the answer would be
> "other program's --porcelain mode", as sending them to stdout would
> be one of the things that make the output easier for programs to
> parse, so it does sound like very much in the same spirit as "git
> status --porcelain" where its output format gets tweaked to be more
> machine friendly.
> 
> The output with "--porcelain" option enabled tend to be less human
> friendly and the distinction between Porcelain (for humans) and
> plumbing (for scripts) is reversed in the use of the word there---it
> started as "this is the option for those who write Porcelain
> commands to use", but still it is not a very good name for the
> option.

Ah, thanks for explaining where this reversed meaning actually comes from. I really wondered why we use "porcelain" in preexisting code to reflect machine-parseable interface, but that explanation does make sense.

> I am perfectly OK if the plan is to uniformly use --output-format
> (or something equally more descriptive) and migrate and deprecate
> the "--porcelain" option away from existing commands.

I'm still not quite clear on where the consensus lies now. Personally, I think that both `--format` and `--output-format` work well and are a bit more descriptive than simply `--porcelain`, and wouldn't mind also migrating other binaries to use either of them.

Furthermore, I think that `--[output-]format` has the advantage that you don't need to handle priorities or mutual exclusion of different options that all apply to the reference format. To a user, it is not immediately obvious what `git fetch --format=compact --porcelain` would do, and which of both options ultimately get respected. But that's likely only true for future commands, because any migration would create the same kind of ambiguity for preexisting commands.

If we were to also migrate preexisting code to use `--[output-]format` then I'd argue that `--output-format` is likely the better name, mostly because it is less likely to be ambiguous compared to `--format`. The latter could e.g. easily confused with `--object-format`.

So I think I'll stick with `--output-format` for the time being.
Patrick
Jacob Keller· Apr 27, 2023, 19:46 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

On 4/27/2023 3:58 AM, Patrick Steinhardt wrote:
Show 16 quoted lines
> On Wed, Apr 26, 2023 at 01:23:12PM -0700, Junio C Hamano wrote:
> Furthermore, I think that `--[output-]format` has the advantage that you
> don't need to handle priorities or mutual exclusion of different options
> that all apply to the reference format. To a user, it is not immediately
> obvious what `git fetch --format=compact --porcelain` would do, and
> which of both options ultimately get respected. But that's likely only
> true for future commands, because any migration would create the same
> kind of ambiguity for preexisting commands.
> 
> If we were to also migrate preexisting code to use `--[output-]format`
> then I'd argue that `--output-format` is likely the better name, mostly
> because it is less likely to be ambiguous compared to `--format`. The
> latter could e.g. easily confused with `--object-format`.
> 
> So I think I'll stick with `--output-format` for the time being.
> 

I agree. I think using --output-format and migrating existing commands that have --porcelain to use --output-format is good. I'm not sure whether to keep using --output-format=porcelain or whether to use a different term there.

I definitely think avoiding confusion with --format is good, and I don't think its too much of a burden to have the longer --output-format as the option name.

Thanks, Jake

> Patrick
Glen Choo· Apr 27, 2023, 22:49 UTC · re: Junio C Hamano · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

Junio C Hamano <gitster@pobox.com> writes:
Show 25 quoted lines
>> We had some discussion during review club about this, where the idea of
>> using "--porcelain" came up because many commands use that when
>> switching into a machine readable format.
>>
>> In addition, this format not only changes the output but also moves it
>> from being on stderr to stdout, which is a hint that the intended usage
>> of the command is now a little different.
>
> A little different from what?  I do not think the answer would be
> "other program's --porcelain mode", as sending them to stdout would
> be one of the things that make the output easier for programs to
> parse, so it does sound like very much in the same spirit as "git
> status --porcelain" where its output format gets tweaked to be more
> machine friendly.
>
> The output with "--porcelain" option enabled tend to be less human
> friendly and the distinction between Porcelain (for humans) and
> plumbing (for scripts) is reversed in the use of the word there---it
> started as "this is the option for those who write Porcelain
> commands to use", but still it is not a very good name for the
> option.
>
> I am perfectly OK if the plan is to uniformly use --output-format
> (or something equally more descriptive) and migrate and deprecate
> the "--porcelain" option away from existing commands.

I agree that --porcelain is a confusing name that would be nice to deprecate, but I don't think --output-format captures all of the intent of "operate in a machine-friendly mode instead of a human-friendly one". Unfortunately, if we had picked --plumbing from the beginning, I doubt we would be having this discussion today :/

E.g. machines (Unix ones at least?) like to have output on stdout and to be able to request NUL-terminated output. It's unfortunate that if we don't piggyback onto --output-format, we run into option precedence problems (like Patrick mentioned), but I'd find it more confusing that --output-format=[porcelain|full|compact] don't behave the same way.

I don't think this puts us in a better spot with regards to option precedence either. Consider:

  git fetch --output-format=full -z <...>

The only way to respect both options is to have -z affect the human-readable output, which isn't the end of the world, but it seems unnecessary.

Perhaps something like --machine instead?
Junio C Hamano· Apr 26, 2023, 20:24 UTC · re: Jacob Keller · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

Jacob Keller <jacob.e.keller@intel.com> writes:
Show 13 quoted lines
>> Yeah, I'd be perfectly happy to rename this to `--format=porcelain`.
>> I'll wait for the Review Club that discusses this patch set tomorrow and
>> will send a new version with that change afterwards if nobody disagrees.
>> 
>> Patrick
>
> We had some discussion during review club about this, where the idea of
> using "--porcelain" came up because many commands use that when
> switching into a machine readable format.
>
> In addition, this format not only changes the output but also moves it
> from being on stderr to stdout, which is a hint that the intended usage
> of the command is now a little different.

A little different from what? I do not think the answer would be "other program's --porcelain mode", as sending them to stdout would be one of the things that make the output easier for programs to parse, so it does sound like very much in the same spirit as "git status --porcelain" where its output format gets tweaked to be more machine friendly.

The output with "--porcelain" option enabled tend to be less human friendly and the distinction between Porcelain (for humans) and plumbing (for scripts) is reversed in the use of the word there---it started as "this is the option for those who write Porcelain commands to use", but still it is not a very good name for the option.

I am perfectly OK if the plan is to uniformly use --output-format (or something equally more descriptive) and migrate and deprecate the "--porcelain" option away from existing commands.

Thanks.
Glen Choo· Apr 26, 2023, 18:54 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

Hi Patrick!

Thanks for the pleasant read! I thought this was a great topic for Review Club. It's too bad that we missed you, but we post all relevant feedback here anyway.

Nevertheless, if you'd like to see the meeting notes, you can find them at:

https://docs.google.com/document/d/14L8BAumGTpsXpjDY8VzZ4rRtpAjuGrFSRqn3stCuS_w/edit
Patrick Steinhardt <ps@pks.im> writes:
Show 20 quoted lines
> Parsing the output of fetches is mostly impossible. It prettifies
> reference names that are about to be updated, doesn't print the old and
> new object IDs the refs are being updated from and to, and prints all of
> that information in nice columns. In short, it is designed to be read by
> humans rather than machines.
>
> This makes it hard to use in a script way though, e.g. to learn about
> which references actually have been updated or which have not been
> updated. This patch series intends to fix that by introducing a new
> machine-parseable interface:
>
> ```
> $ git fetch --output-format=porcelain --no-progress
>   fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
> * 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
> * 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
> * 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
> * 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
> * 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
> ```

Having machine-parseable output seems like an obviously good goal to me. The finer points of the interface and output format are worth discussing. I'll do so in the patches themselves.

Overall, I found the series very well-structured and easy to follow along. Thanks.

Glen Choo· Apr 26, 2023, 21:14 UTC · re: Glen Choo · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

Glen Choo <chooglen@google.com> writes:
Show 10 quoted lines
> Hi Patrick!
>
> Thanks for the pleasant read! I thought this was a great topic for
> Review Club. It's too bad that we missed you, but we post all relevant
> feedback here anyway.
>
> Nevertheless, if you'd like to see the meeting notes, you can find them
> at:
>
> https://docs.google.com/document/d/14L8BAumGTpsXpjDY8VzZ4rRtpAjuGrFSRqn3stCuS_w/edit

It seems like Jacob Keller and I have mostly the same comments (since we did both go to the same Review Club, after all :)), so feel free to respond to only Jacob's messages and ignore mine (where the messages overlap) - I'll catch up on those conversations.

Jacob Keller· Apr 26, 2023, 19:17 UTC · re: Patrick Steinhardt · lore

Re: [PATCH 0/8] fetch: introduce machine-parseable output

On 4/19/2023 5:31 AM, Patrick Steinhardt wrote:
Show 11 quoted lines
> 
> ```
> $ git fetch --output-format=porcelain --no-progress
>   fff5a5e7f528b2ed2c335991399a766c2cf01103 af67688dca57999fd848f051eeea1d375ba546b2 refs/remotes/origin/master
> * 0000000000000000000000000000000000000000 e046fe5a36a970bc14fbfbcb2074a48776f6b671 refs/remotes/origin/x86-rep-insns
> * 0000000000000000000000000000000000000000 bb81ed6862b864c9eb99447f04d49a84ecb647e5 refs/tags/v6.3-rc4
> * 0000000000000000000000000000000000000000 83af7b1468c0dca86b4dc9e43e73bfa4f38d9637 refs/tags/v6.3-rc5
> * 0000000000000000000000000000000000000000 ab3affb8ed84f68638162fe7e6fd4055e15bff5b refs/tags/v6.3-rc6
> * 0000000000000000000000000000000000000000 1c8c28415e8743368a2b800520a6dd0b22ee6ec2 refs/tags/v6.3-rc7
> ```
> 

One thing that the standard output (maybe with --progress?) shows to stderr is which remote is being fetched and what URL was fetched.

This seems like useful data to include in a machine readable format. It wasn't clear if such output would still be made and whether it would go to stderr or stdout, nor whether that existing output was machine readable.

Obviously you can somewhat infer remotes based on the refs/remotes, but that doesn't include other refs like tags.

When fetching from multiple reports (--all or explicitly asking), I wonder if it makes sense to include some lines to differentiate where each block of updates for a given remote starts and ends?

Thanks, Jake

Patrick Steinhardt· Apr 27, 2023, 11:13 UTC · re: Patrick Steinhardt · lore

[PATCH v2 0/8] fetch: introduce machine-parseable output

Hi,

this is the second version of my patch series that introduces a machine-parseable output.

Changes compared to v1:
    - Patch 3/8: I've reworded the commit message to hopefully be more
      straight-forward. I've also amended tests so that we don't only
      test with `--dry-run`, but also without.
    - Patch 6/8: I've also moved the `all` and `multiple` variables from
      their global scope into `cmd__fetch`.
    - Patch 7/8: Fixed a bug where `--output-format=` wasn't honored for
      child processes when doing multi-remote fetches. Furthermore, I've
      unified parsing of the actual format so that we don't have to
      repeat it thrice.
    - Patch 8/8: Added a note to the commit message that tries to argue
      why I didn't add remote information to the interface. I'm still
      open to change this if you disagree with my reasoning here.

Thanks a bunch for all the feedback received so far, really appreciate it!

Patrick
Patrick Steinhardt (8):
  fetch: split out tests for output format
  fetch: add a test to exercise invalid output formats
  fetch: fix missing from-reference when fetching HEAD:foo
  fetch: introduce `display_format` enum
  fetch: move display format parsing into main function
  fetch: move option related variables into main function
  fetch: introduce new `--output-format` option
  fetch: introduce machine-parseable "porcelain" output format
 Documentation/config/fetch.txt  |   4 +-
 Documentation/fetch-options.txt |   5 +
 Documentation/git-fetch.txt     |  17 +-
 builtin/fetch.c                 | 427 ++++++++++++++++++++------------
 t/t5510-fetch.sh                |  53 ----
 t/t5574-fetch-output.sh         | 237 ++++++++++++++++++
 6 files changed, 521 insertions(+), 222 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Range-diff against v1:
1:  0d0d50d14c = 1:  0d0d50d14c fetch: split out tests for output format
2:  29d2c58914 = 2:  29d2c58914 fetch: add a test to exercise invalid output formats
3:  596e12f03a ! 3:  d1fb6eeae7 fetch: fix missing from-reference when fetching HEAD:foo
    @@ Metadata
      ## Commit message ##
         fetch: fix missing from-reference when fetching HEAD:foo
     
    -    When displaying reference updates, we print a line that looks similar to
    -    the following:
    +    `store_updated_refs()` parses the remote reference for two purposes:
    +
    +        - It gets used as a note when writing FETCH_HEAD.
    +
    +        - It is passed through to `display_ref_update()` to display
    +          updated references in the following format:
    +
    +          ```
    +           * branch               master          -> master
    +          ```
    +
    +    In most cases, the parsed remote reference is the prettified reference
    +    name and can thus be used for both cases. But if the remote reference is
    +    HEAD, the parsed remote reference becomes empty. This is intended when
    +    we write the FETCH_HEAD, where we skip writing the note in that case.
    +    But it is not intended when displaying the updated references and would
    +    cause us to miss the left-hand side of the displayed reference update:
     
         ```
    -     * branch               master          -> master
    -    ```
    -
    -    The "branch" bit changes depending on what kind of reference we're
    -    updating, while both of the right-hand references are computed by
    -    stripping well-known prefixes like "refs/heads/" or "refs/tags".
    -
    -    The logic is kind of intertwined though and not easy to follow: we
    -    precompute both the kind (e.g. "branch") and the what, which is the
    -    abbreviated remote reference name, in `store_updated_refs()` and then
    -    pass it down the call chain to `display_ref_update()`.
    -
    -    There is a set of different cases here:
    -
    -        - When the remote reference name is "HEAD" we assume no kind and
    -          will thus instead print "[new ref]". We keep what at the empty
    -          string.
    -
    -        - When the remote reference name has a well-known prefix then the
    -          kind would be "branch", "tag" or "remote-tracking branch". The
    -          what is the reference with the well-known prefix stripped and in
    -          fact matches the output that `prettify_refname()` would return.
    -
    -        - Otherwise, we'll again assume no kind and keep the what set to the
    -          fully qualified reference name.
    -
    -    Now there is a bug with the first case here, where the remote reference
    -    name is "HEAD". As noted, "what" will be set to the empty string. And
    -    that seems to be intentional because we also use this information to
    -    update the FETCH_HEAD, and in case we're updating HEAD we seemingly
    -    don't want to append that to our FETCH_HEAD value.
    -
    -    But as mentioned, we also use this value to display reference updates.
    -    And while the call to `display_ref_update()` correctly figures out that
    -    we meant "HEAD" when `what` is empty, the call to `update_local_ref()`
    -    doesn't. `update_local_ref()` will then call `display_ref_update()` with
    -    the empty string and cause the following broken output:
    -
    -    ```
    -    $ git fetch --dry-run origin HEAD:foo
    +    $ git fetch origin HEAD:foo
         From https://github.com/git/git
          * [new ref]                          -> foo
         ```
     
         The HEAD string is clearly missing from the left-hand side of the arrow,
    -    which is further stressed by the point that the following commands work
    -    as expected:
    +    which is further stressed by the point that the following commands show
    +    the left-hand side as expected:
     
         ```
    -    $ git fetch --dry-run origin HEAD
    +    $ git fetch origin HEAD
         From https://github.com/git/git
          * branch                  HEAD       -> FETCH_HEAD
     
    -    $ git fetch --dry-run origin master
    +    $ git fetch origin master
         From https://github.com/git/git
          * branch                  master     -> FETCH_HEAD
          * branch                  master     -> origin/master
         ```
     
    -    Fix this bug by instead unconditionally passing the full reference name
    -    to `display_ref_update()` which learns to call `prettify_refname()` on
    -    it. This does fix the above bug and is otherwise functionally the same
    -    as `prettify_refname()` would only ever strip the well-known prefixes
    -    just as intended. So at the same time, this also simplifies the code a
    -    bit.
    +    The logic of how we compute the remote reference name that we ultimately
    +    pass to `display_ref_update()` is not easy to follow. There are three
    +    different cases here:
    +
    +        - When the remote reference name is "HEAD" we set the remote
    +          reference name to the empty string. This is the case that causes
    +          the bug to occur, where we would indeed want to print "HEAD"
    +          instead of the empty string. This is what `prettify_refname()`
    +          would return.
    +
    +        - When the remote reference name has a well-known prefix then we
    +          strip this prefix. This matches what `prettify_refname()` does.
    +
    +        - Otherwise, we keep the fully qualified reference name. This also
    +          matches what `prettify_refname()` does.
    +
    +    As the return value of `prettify_refname()` would do the correct thing
    +    for us in all three cases, we can fix the bug by passing through the
    +    full remote reference name to `display_ref_update()`, which learns to
    +    call `prettify_refname()`. At the same time, this also simplifies the
    +    code a bit.
     
         Note that this patch also changes formatting of the block that computes
         the "kind" and "what" variables. This is done on purpose so that it is
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +	EOF
     +	test_cmp expect actual &&
     +
    ++	git -C head fetch origin HEAD >actual 2>&1 &&
    ++	test_cmp expect actual &&
    ++
     +	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
     +	cat >expect <<-EOF &&
     +	From $(test-tool path-utils real_path .)/.
     +	 * [new ref]         HEAD       -> foo
     +	EOF
    ++	test_cmp expect actual &&
    ++
    ++	git -C head fetch origin HEAD:foo >actual 2>&1 &&
     +	test_cmp expect actual
     +'
     +
4:  8571363be1 = 4:  b545bf8bb9 fetch: introduce `display_format` enum
5:  d98c3ee0ce = 5:  4990d35998 fetch: move display format parsing into main function
6:  640a8840e1 ! 6:  cfe84129ab fetch: move option related variables into main function
    @@ Commit message
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## builtin/fetch.c ##
    -@@ builtin/fetch.c: static int all, append, dry_run, force, keep, multiple, update_head_ok;
    +@@ builtin/fetch.c: static int fetch_prune_tags_config = -1; /* unspecified */
    + static int prune_tags = -1; /* unspecified */
    + #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
    + 
    +-static int all, append, dry_run, force, keep, multiple, update_head_ok;
    ++static int append, dry_run, force, keep, update_head_ok;
      static int write_fetch_head = 1;
      static int verbosity, deepen_relative, set_upstream, refetch;
      static int progress = -1;
    @@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const cha
      	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
      	struct string_list list = STRING_LIST_INIT_DUP;
      	struct remote *remote = NULL;
    ++	int all = 0, multiple = 0;
      	int result = 0;
      	int prune_tags_ok = 1;
     +	int enable_auto_gc = 1;
7:  3b2cad066a ! 7:  0335e5eeb4 fetch: introduce new `--output-format` option
    @@ Commit message
         current mechanism feels a little bit indirect and rigid.
     
         Introduce a new `--output-format` option that allows the user to change
    -    the desired output format more directly.
    +    the desired format more directly.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    @@ Documentation/fetch-options.txt: linkgit:git-config[1].
      	Write the list of remote refs fetched in the `FETCH_HEAD`
     
      ## builtin/fetch.c ##
    +@@ builtin/fetch.c: enum display_format {
    + 	DISPLAY_FORMAT_UNKNOWN = 0,
    + 	DISPLAY_FORMAT_FULL,
    + 	DISPLAY_FORMAT_COMPACT,
    ++	DISPLAY_FORMAT_MAX,
    ++};
    ++
    ++static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
    ++	NULL,
    ++	"full",
    ++	"compact",
    + };
    + 
    + struct display_state {
    +@@ builtin/fetch.c: static int fetch_finished(int result, struct strbuf *out,
    + 	return 0;
    + }
    + 
    +-static int fetch_multiple(struct string_list *list, int max_children)
    ++static int fetch_multiple(struct string_list *list, int max_children,
    ++			  enum display_format format)
    + {
    + 	int i, result = 0;
    + 	struct strvec argv = STRVEC_INIT;
    +@@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_children)
    + 		     "--no-write-commit-graph", NULL);
    + 	add_options_to_argv(&argv);
    + 
    ++	if (format != DISPLAY_FORMAT_UNKNOWN)
    ++		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
    ++
    + 	if (max_children != 1 && list->nr != 1) {
    + 		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
    + 		const struct run_process_parallel_opts opts = {
     @@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv,
      	return exit_code;
      }
      
    ++static enum display_format parse_display_format(const char *format)
    ++{
    ++	for (int i = 0; i < ARRAY_SIZE(display_formats); i++)
    ++		if (display_formats[i] && !strcmp(display_formats[i], format))
    ++			return i;
    ++	return DISPLAY_FORMAT_UNKNOWN;
    ++}
    ++
     +static int opt_parse_output_format(const struct option *opt, const char *arg, int unset)
     +{
    -+	enum display_format *format = opt->value;
    ++	enum display_format *format = opt->value, parsed;
    ++
     +	if (unset || !arg)
     +		return 1;
    -+	else if (!strcmp(arg, "full"))
    -+		*format = DISPLAY_FORMAT_FULL;
    -+	else if (!strcmp(arg, "compact"))
    -+		*format = DISPLAY_FORMAT_COMPACT;
    -+	else
    ++
    ++	parsed = parse_display_format(arg);
    ++	if (parsed == DISPLAY_FORMAT_UNKNOWN)
     +		return error(_("unsupported output format '%s'"), arg);
    ++	*format = parsed;
     +
     +	return 0;
     +}
    @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
      			 N_("write fetched references to the FETCH_HEAD file")),
      		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 		const char *format = "full";
    + 
    + 		git_config_get_string_tmp("fetch.output", &format);
    +-		if (!strcasecmp(format, "full"))
    +-			display_format = DISPLAY_FORMAT_FULL;
    +-		else if (!strcasecmp(format, "compact"))
    +-			display_format = DISPLAY_FORMAT_COMPACT;
    +-		else
    ++
    ++		display_format = parse_display_format(format);
    ++		if (display_format == DISPLAY_FORMAT_UNKNOWN)
    + 			die(_("invalid value for '%s': '%s'"),
    + 			    "fetch.output", format);
    + 	}
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 			max_children = fetch_parallel_config;
    + 
    + 		/* TODO should this also die if we have a previous partial-clone? */
    +-		result = fetch_multiple(&list, max_children);
    ++		result = fetch_multiple(&list, max_children, display_format);
    + 	}
    + 
    + 
     
      ## t/t5574-fetch-output.sh ##
     @@ t/t5574-fetch-output.sh: test_expect_success 'fetch with invalid output format configuration' '
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch aligned output' '
      	cat >expect <<-\EOF &&
      	main       -> origin/*
      	extraaa    -> *
    +@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
    + 	test_cmp expect actual
    + '
    + 
    ++test_expect_success 'fetch compact output with multiple remotes' '
    ++	test_when_finished "rm -rf compact-cfg compact-cli" &&
    ++
    ++	git clone . compact-cli &&
    ++	git -C compact-cli remote add second-remote "$PWD" &&
    ++	git clone . compact-cfg &&
    ++	git -C compact-cfg remote add second-remote "$PWD" &&
    ++	test_commit multi-commit &&
    ++
    ++	git -C compact-cfg -c fetch.output=compact fetch --all >actual-cfg 2>&1 &&
    ++	git -C compact-cli fetch --output-format=compact --all >actual-cli 2>&1 &&
    ++	test_cmp actual-cfg actual-cli &&
    ++
    ++	grep -e "->" actual-cfg | cut -c 22- >actual &&
    ++	cat >expect <<-\EOF &&
    ++	main         -> origin/*
    ++	multi-commit -> *
    ++	main       -> second-remote/*
    ++	EOF
    ++	test_cmp expect actual
    ++'
    ++
    + test_expect_success 'fetch output with HEAD and --dry-run' '
    + 	test_when_finished "rm -rf head" &&
    + 	git clone . head &&
8:  301138da03 ! 8:  d7c1bc1a80 fetch: introduce machine-parseable "porcelain" output format
    @@ Commit message
         so that other data printed to stderr, like error messages or progress
         meters, don't interfere with the parseable data.
     
    +    A notable ommission here is that the output format does not include the
    +    remote from which a reference was fetched, which might be important
    +    information especially in the context of multi-remote fetches. But as
    +    such a format would require us to print the remote for every single
    +    reference update due to parallelizable fetches it feels wasteful for the
    +    most likely usecase, which is when fetching from a single remote. If
    +    usecases come up for this in the future though it is easy enough to add
    +    a new "porcelain-v2" format that adds this information.
    +
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## Documentation/config/fetch.txt ##
    @@ builtin/fetch.c: enum display_format {
      	DISPLAY_FORMAT_FULL,
      	DISPLAY_FORMAT_COMPACT,
     +	DISPLAY_FORMAT_PORCELAIN,
    + 	DISPLAY_FORMAT_MAX,
    + };
    + 
    +@@ builtin/fetch.c: static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
    + 	NULL,
    + 	"full",
    + 	"compact",
    ++	"porcelain",
      };
      
      struct display_state {
    @@ builtin/fetch.c: static int prune_refs(struct display_state *display_state,
      					   summary_width);
      			warn_dangling_symref(stderr, dangling_msg, ref->name);
      		}
    -@@ builtin/fetch.c: static int opt_parse_output_format(const struct option *opt, const char *arg, in
    - 		*format = DISPLAY_FORMAT_FULL;
    - 	else if (!strcmp(arg, "compact"))
    - 		*format = DISPLAY_FORMAT_COMPACT;
    -+	else if (!strcmp(arg, "porcelain"))
    -+		*format = DISPLAY_FORMAT_PORCELAIN;
    - 	else
    - 		return error(_("unsupported output format '%s'"), arg);
    - 
    -@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    - 			display_format = DISPLAY_FORMAT_FULL;
    - 		else if (!strcasecmp(format, "compact"))
    - 			display_format = DISPLAY_FORMAT_COMPACT;
    -+		else if (!strcasecmp(format, "porcelain"))
    -+			display_format = DISPLAY_FORMAT_PORCELAIN;
    - 		else
    - 			die(_("invalid value for '%s': '%s'"),
    - 			    "fetch.output", format);
     
      ## t/t5574-fetch-output.sh ##
    -@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
    +@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output with multiple remotes' '
      	test_cmp expect actual
      '
      
-- 
2.40.1
Patrick Steinhardt· Apr 27, 2023, 11:13 UTC · re: Patrick Steinhardt · lore

[PATCH v2 1/8] fetch: split out tests for output format

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Show changes to 2 files +63 −53

t/t5510-fetch.sh, t/t5574-fetch-output.sh

diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index dc44da9c79..4f289063ce 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
new file mode 100755
index 0000000000..f91b654d38
--- /dev/null
+++ b/t/t5574-fetch-output.sh
@@ -0,0 +1,63 @@
+#!/bin/sh
+
+test_description='git fetch output format'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+
+test_expect_success 'fetch aligned output' '
+	git clone . full-output &&
+	test_commit looooooooooooong-tag &&
+	(
+		cd full-output &&
+		git -c fetch.output=full fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main                 -> origin/main
+	looooooooooooong-tag -> looooooooooooong-tag
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'fetch compact output' '
+	git clone . compact &&
+	test_commit extraaa &&
+	(
+		cd compact &&
+		git -c fetch.output=compact fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main       -> origin/*
+	extraaa    -> *
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success '--no-show-forced-updates' '
+	mkdir forced-updates &&
+	(
+		cd forced-updates &&
+		git init &&
+		test_commit 1 &&
+		test_commit 2
+	) &&
+	git clone forced-updates forced-update-clone &&
+	git clone forced-updates no-forced-update-clone &&
+	git -C forced-updates reset --hard HEAD~1 &&
+	(
+		cd forced-update-clone &&
+		git fetch --show-forced-updates origin 2>output &&
+		test_i18ngrep "(forced update)" output
+	) &&
+	(
+		cd no-forced-update-clone &&
+		git fetch --no-show-forced-updates origin 2>output &&
+		test_i18ngrep ! "(forced update)" output
+	)
+'
+
+test_done
-- 
2.40.1
SZEDER Gábor· Apr 29, 2023, 17:34 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 1/8] fetch: split out tests for output format

On Thu, Apr 27, 2023 at 01:13:08PM +0200, Patrick Steinhardt wrote:
Show 16 quoted lines
> We're about to introduce a new porcelain mode for the output of
> git-fetch(1). As part of that we'll be introducing a set of new tests
> that only relate to the output of this command.
> 
> Split out tests that exercise the output format of git-fetch(1) so that
> it becomes easier to verify this functionality as a standalone unit. As
> the tests assume that the default branch is called "main" we set up the
> corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable
> accordingly.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  t/t5510-fetch.sh        | 53 ----------------------------------
>  t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 63 insertions(+), 53 deletions(-)
>  create mode 100755 t/t5574-fetch-output.sh
Show 21 quoted lines
> diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
> new file mode 100755
> index 0000000000..f91b654d38
> --- /dev/null
> +++ b/t/t5574-fetch-output.sh
> @@ -0,0 +1,63 @@
> +#!/bin/sh
> +
> +test_description='git fetch output format'
> +
> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
> +
> +. ./test-lib.sh
> +
> +test_expect_success 'fetch aligned output' '
> +	git clone . full-output &&
> +	test_commit looooooooooooong-tag &&
> +	(
> +		cd full-output &&
> +		git -c fetch.output=full fetch origin >actual 2>&1 &&

Why is that 2>&1 redirection used here? If the output the test case is interested in goes to the command's standard output, then it's unnecessary. However, if it goes to standard error, then why is standard output redirected as well?

I understand that this patch just moves existing test cases around as-is, so this is not something you introduced, but I point it out here, because later patches of this series add several new test cases following this anti-pattern.

Since this series creates a new test script, perhaps this might be the right time to clean this up.

Show 38 quoted lines
> +		grep -e "->" actual | cut -c 22- >../actual
> +	) &&
> +	cat >expect <<-\EOF &&
> +	main                 -> origin/main
> +	looooooooooooong-tag -> looooooooooooong-tag
> +	EOF
> +	test_cmp expect actual
> +'
> +
> +test_expect_success 'fetch compact output' '
> +	git clone . compact &&
> +	test_commit extraaa &&
> +	(
> +		cd compact &&
> +		git -c fetch.output=compact fetch origin >actual 2>&1 &&
> +		grep -e "->" actual | cut -c 22- >../actual
> +	) &&
> +	cat >expect <<-\EOF &&
> +	main       -> origin/*
> +	extraaa    -> *
> +	EOF
> +	test_cmp expect actual
> +'
> +
> +test_expect_success '--no-show-forced-updates' '
> +	mkdir forced-updates &&
> +	(
> +		cd forced-updates &&
> +		git init &&
> +		test_commit 1 &&
> +		test_commit 2
> +	) &&
> +	git clone forced-updates forced-update-clone &&
> +	git clone forced-updates no-forced-update-clone &&
> +	git -C forced-updates reset --hard HEAD~1 &&
> +	(
> +		cd forced-update-clone &&
> +		git fetch --show-forced-updates origin 2>output &&
Oh, look, there are some good examples to follow here as well :)
Show 13 quoted lines
> +		test_i18ngrep "(forced update)" output
> +	) &&
> +	(
> +		cd no-forced-update-clone &&
> +		git fetch --no-show-forced-updates origin 2>output &&
> +		test_i18ngrep ! "(forced update)" output
> +	)
> +'
> +
> +test_done
> -- 
> 2.40.1
> 
Patrick Steinhardt· May 3, 2023, 11:21 UTC · re: SZEDER Gábor · lore

Re: [PATCH v2 1/8] fetch: split out tests for output format

On Sat, Apr 29, 2023 at 07:34:48PM +0200, SZEDER Gábor wrote:
Show 53 quoted lines
> On Thu, Apr 27, 2023 at 01:13:08PM +0200, Patrick Steinhardt wrote:
> > We're about to introduce a new porcelain mode for the output of
> > git-fetch(1). As part of that we'll be introducing a set of new tests
> > that only relate to the output of this command.
> > 
> > Split out tests that exercise the output format of git-fetch(1) so that
> > it becomes easier to verify this functionality as a standalone unit. As
> > the tests assume that the default branch is called "main" we set up the
> > corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable
> > accordingly.
> > 
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >  t/t5510-fetch.sh        | 53 ----------------------------------
> >  t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 63 insertions(+), 53 deletions(-)
> >  create mode 100755 t/t5574-fetch-output.sh
> 
> 
> > diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
> > new file mode 100755
> > index 0000000000..f91b654d38
> > --- /dev/null
> > +++ b/t/t5574-fetch-output.sh
> > @@ -0,0 +1,63 @@
> > +#!/bin/sh
> > +
> > +test_description='git fetch output format'
> > +
> > +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
> > +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
> > +
> > +. ./test-lib.sh
> > +
> > +test_expect_success 'fetch aligned output' '
> > +	git clone . full-output &&
> > +	test_commit looooooooooooong-tag &&
> > +	(
> > +		cd full-output &&
> > +		git -c fetch.output=full fetch origin >actual 2>&1 &&
> 
> Why is that 2>&1 redirection used here?
> If the output the test case is interested in goes to the command's
> standard output, then it's unnecessary.  However, if it goes to
> standard error, then why is standard output redirected as well?
> 
> I understand that this patch just moves existing test cases around
> as-is, so this is not something you introduced, but I point it out
> here, because later patches of this series add several new test cases
> following this anti-pattern.
> 
> Since this series creates a new test script, perhaps this might be the
> right time to clean this up.

I feel like this patch series is big enough on its own already, so for now I'd like to refrain from touching up the old tests. But you've got a good point here, and I'll make sure that any new tests don't followw the same pattern.

That being said, I think we should keep on testing both stdout and stderr. The tests here are explicitly about the output format, so it does make sense to verify that both of them match what we expect. We should treat them as separate things though.

Patrick
Patrick Steinhardt· Apr 27, 2023, 11:13 UTC · re: Patrick Steinhardt · lore

[PATCH v2 2/8] fetch: add a test to exercise invalid output formats

Add a testcase that exercises the logic when an invalid output format is passed via the `fetch.output` configuration.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5574-fetch-output.sh | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
Show changes to t/t5574-fetch-output.sh +17 −0
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index f91b654d38..0e45c27007 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -7,6 +7,23 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_cmp expect actual &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
-- 
2.40.1
Patrick Steinhardt· Apr 27, 2023, 11:13 UTC · re: Patrick Steinhardt · lore

[PATCH v2 3/8] fetch: fix missing from-reference when fetching HEAD:foo

`store_updated_refs()` parses the remote reference for two purposes:
    - It gets used as a note when writing FETCH_HEAD.
    - It is passed through to `display_ref_update()` to display
      updated references in the following format:
      ```
       * branch               master          -> master
      ```

In most cases, the parsed remote reference is the prettified reference name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case. But it is not intended when displaying the updated references and would cause us to miss the left-hand side of the displayed reference update:

```
$ git fetch origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands show the left-hand side as expected:

```
$ git fetch origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD
$ git fetch origin master
From https://github.com/git/git
 * branch                  master     -> FETCH_HEAD
 * branch                  master     -> origin/master
```

The logic of how we compute the remote reference name that we ultimately pass to `display_ref_update()` is not easy to follow. There are three different cases here:

    - When the remote reference name is "HEAD" we set the remote
      reference name to the empty string. This is the case that causes
      the bug to occur, where we would indeed want to print "HEAD"
      instead of the empty string. This is what `prettify_refname()`
      would return.
    - When the remote reference name has a well-known prefix then we
      strip this prefix. This matches what `prettify_refname()` does.
    - Otherwise, we keep the fully qualified reference name. This also
      matches what `prettify_refname()` does.

As the return value of `prettify_refname()` would do the correct thing for us in all three cases, we can fix the bug by passing through the full remote reference name to `display_ref_update()`, which learns to call `prettify_refname()`. At the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c         | 37 +++++++++++++++++++------------------
 t/t5574-fetch-output.sh | 25 +++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 18 deletions(-)
Show changes to 2 files +44 −18

builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index c310d89878..7c64f0c562 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index 0e45c27007..b9dcdade63 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -54,6 +54,31 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch origin HEAD >actual 2>&1 &&
+	test_cmp expect actual &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch origin HEAD:foo >actual 2>&1 &&
+	test_cmp expect actual
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
-- 
2.40.1
Glen Choo· Apr 27, 2023, 17:26 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 3/8] fetch: fix missing from-reference when fetching HEAD:foo

For v2, I've only read through this patch for now. I'm still formulating my thoughts on the output format and the CLI option name.

Patrick Steinhardt <ps@pks.im> writes:
Show 25 quoted lines
> `store_updated_refs()` parses the remote reference for two purposes:
>
>     - It gets used as a note when writing FETCH_HEAD.
>
>     - It is passed through to `display_ref_update()` to display
>       updated references in the following format:
>
>       ```
>        * branch               master          -> master
>       ```
>
> In most cases, the parsed remote reference is the prettified reference
> name and can thus be used for both cases. But if the remote reference is
> HEAD, the parsed remote reference becomes empty. This is intended when
> we write the FETCH_HEAD, where we skip writing the note in that case.
> But it is not intended when displaying the updated references and would
> cause us to miss the left-hand side of the displayed reference update:
>
> [...]
>
> As the return value of `prettify_refname()` would do the correct thing
> for us in all three cases, we can fix the bug by passing through the
> full remote reference name to `display_ref_update()`, which learns to
> call `prettify_refname()`. At the same time, this also simplifies the
> code a bit.

This rewritten section explains the bug and the fix very nicely :) Thanks!

Show 21 quoted lines
> diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
> index 0e45c27007..b9dcdade63 100755
> --- a/t/t5574-fetch-output.sh
> +++ b/t/t5574-fetch-output.sh
> @@ -54,6 +54,31 @@ test_expect_success 'fetch compact output' '
>  	test_cmp expect actual
>  '
>  
> +test_expect_success 'fetch output with HEAD and --dry-run' '
> +	test_when_finished "rm -rf head" &&
> +	git clone . head &&
> +
> +	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
> +	cat >expect <<-EOF &&
> +	From $(test-tool path-utils real_path .)/.
> +	 * branch            HEAD       -> FETCH_HEAD
> +	EOF
> +	test_cmp expect actual &&
> +
> +	git -C head fetch origin HEAD >actual 2>&1 &&
> +	test_cmp expect actual &&

Maybe rename the test to just 'fetch output with HEAD'? The 'and --dry-run' part seems completely obsolete.

Jacob Keller· Apr 27, 2023, 19:49 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 3/8] fetch: fix missing from-reference when fetching HEAD:foo

On 4/27/2023 4:13 AM, Patrick Steinhardt wrote:
Show 9 quoted lines
> index 0e45c27007..b9dcdade63 100755
> --- a/t/t5574-fetch-output.sh
> +++ b/t/t5574-fetch-output.sh
> @@ -54,6 +54,31 @@ test_expect_success 'fetch compact output' '
>  	test_cmp expect actual
>  '
>  
> +test_expect_success 'fetch output with HEAD and --dry-run' '
> +	test_when_finished "rm -rf head" &&

The test name could drop the --dry-run mention since this now tests both variants.

Everything else looks good on this patch. The new commit message is much better to me now!

Patrick Steinhardt· Apr 27, 2023, 11:13 UTC · re: Patrick Steinhardt · lore

[PATCH v2 4/8] fetch: introduce `display_format` enum

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a `enum display_format` that is more readily extensible.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)
Show changes to builtin/fetch.c +64 −37
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 7c64f0c562..e03fcd1b2f 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -784,7 +790,6 @@ static int refcol_width(const struct ref *ref, int compact_format)
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display foramt %d", display_state->format);
 	}
 }
 
@@ -904,30 +920,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
-- 
2.40.1
Patrick Steinhardt· Apr 27, 2023, 11:13 UTC · re: Patrick Steinhardt · lore

[PATCH v2 5/8] fetch: move display format parsing into main function

Parsing the display format happens inside of `display_state_init()`. As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a output format though that is intended to be parseable by machines, for example inside of a script. In that case it becomes a bit awkward of an interface if you have to call git-fetch(1) with the `fetch.output` config key set. We're thus going to introduce a new `--output-format` switch for git-fetch(1) so that the output format can be configured more directly.

This means we'll have to hook parsing of the display format into the command line options parser. Having the code to determine the actual output format scattered across two different sites is hard to reason about though.

Refactor the code such that callers are expected to pass the display format that is to be used into `display_state_init()`. This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
Show changes to builtin/fetch.c +24 −17
diff --git a/builtin/fetch.c b/builtin/fetch.c
index e03fcd1b2f..bcc156a9ce 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -788,14 +788,13 @@ static int refcol_width(const struct ref *ref, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -812,15 +811,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT: {
@@ -1614,7 +1604,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1700,7 +1691,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2076,7 +2067,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2143,7 +2135,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2155,6 +2147,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	int i;
 	const char *bundle_uri;
+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
@@ -2181,6 +2174,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
+		const char *format = "full";
+
+		git_config_get_string_tmp("fetch.output", &format);
+		if (!strcasecmp(format, "full"))
+			display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(format, "compact"))
+			display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", format);
+	}
+
 	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
 		recurse_submodules = recurse_submodules_cli;
 
@@ -2309,7 +2315,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   display_format);
 	} else {
 		int max_children = max_jobs;
 
-- 
2.40.1
Patrick Steinhardt· Apr 27, 2023, 11:13 UTC · re: Patrick Steinhardt · lore

[PATCH v2 6/8] fetch: move option related variables into main function

The options of git-fetch(1) which we pass to `parse_options()` are declared globally in `builtin/fetch.c`. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into `cmd_fetch()`. Move variables that were previously forced to be declared globally and which are only used by `cmd_fetch()` into function-local scope.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 197 ++++++++++++++++++++++++------------------------
 1 file changed, 100 insertions(+), 97 deletions(-)
Show changes to builtin/fetch.c +100 −97
diff --git a/builtin/fetch.c b/builtin/fetch.c
index bcc156a9ce..97a510649c 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -75,13 +75,12 @@ static int fetch_prune_tags_config = -1; /* unspecified */
 static int prune_tags = -1; /* unspecified */
 #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
 
-static int all, append, dry_run, force, keep, multiple, update_head_ok;
+static int append, dry_run, force, keep, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
@@ -160,92 +153,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2145,13 +2052,109 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
-	int i;
 	const char *bundle_uri;
+	const char *submodule_prefix = "";
 	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
+	int all = 0, multiple = 0;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
-- 
2.40.1
Junio C Hamano· Apr 27, 2023, 21:52 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 6/8] fetch: move option related variables into main function

Patrick Steinhardt <ps@pks.im> writes:
Show 8 quoted lines
> The options of git-fetch(1) which we pass to `parse_options()` are
> declared globally in `builtin/fetch.c`. This means we're forced to use
> global variables for all the options, which is more likely to cause
> confusion than explicitly passing state around.
>
> Refactor the code to move the options into `cmd_fetch()`. Move variables
> that were previously forced to be declared globally and which are only
> used by `cmd_fetch()` into function-local scope.
Very well done.
Patrick Steinhardt· Apr 27, 2023, 11:13 UTC · re: Patrick Steinhardt · lore

[PATCH v2 7/8] fetch: introduce new `--output-format` option

It is only possible to configure the output format that git-fetch(1) uses by setting it via a config key. While this interface may be fine as long as we only have the current "full" and "compact" output formats, where it is unlikely that the user will have to change them regularly. But we're about to introduce a new machine-parseable interface where the current mechanism feels a little bit indirect and rigid.

Introduce a new `--output-format` option that allows the user to change the desired format more directly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/fetch-options.txt |  5 +++
 builtin/fetch.c                 | 48 ++++++++++++++++++----
 t/t5574-fetch-output.sh         | 72 +++++++++++++++++++++++++++------
 3 files changed, 106 insertions(+), 19 deletions(-)
Show changes to 3 files +106 −19

Documentation/fetch-options.txt, builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 622bd84768..654f96f79d 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -78,6 +78,11 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--output-format::
+	Control how ref update status is printed. Valid values are
+	`full` and `compact`. Default value is `full`. See section
+	OUTPUT in linkgit:git-fetch[1] for detail.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 97a510649c..30099b2ac3 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -52,6 +52,13 @@ enum display_format {
 	DISPLAY_FORMAT_UNKNOWN = 0,
 	DISPLAY_FORMAT_FULL,
 	DISPLAY_FORMAT_COMPACT,
+	DISPLAY_FORMAT_MAX,
+};
+
+static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
+	NULL,
+	"full",
+	"compact",
 };
 
 struct display_state {
@@ -1879,7 +1886,8 @@ static int fetch_finished(int result, struct strbuf *out,
 	return 0;
 }
 
-static int fetch_multiple(struct string_list *list, int max_children)
+static int fetch_multiple(struct string_list *list, int max_children,
+			  enum display_format format)
 {
 	int i, result = 0;
 	struct strvec argv = STRVEC_INIT;
@@ -1894,6 +1902,9 @@ static int fetch_multiple(struct string_list *list, int max_children)
 		     "--no-write-commit-graph", NULL);
 	add_options_to_argv(&argv);
 
+	if (format != DISPLAY_FORMAT_UNKNOWN)
+		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
+
 	if (max_children != 1 && list->nr != 1) {
 		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
 		const struct run_process_parallel_opts opts = {
@@ -2050,6 +2061,29 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	return exit_code;
 }
 
+static enum display_format parse_display_format(const char *format)
+{
+	for (int i = 0; i < ARRAY_SIZE(display_formats); i++)
+		if (display_formats[i] && !strcmp(display_formats[i], format))
+			return i;
+	return DISPLAY_FORMAT_UNKNOWN;
+}
+
+static int opt_parse_output_format(const struct option *opt, const char *arg, int unset)
+{
+	enum display_format *format = opt->value, parsed;
+
+	if (unset || !arg)
+		return 1;
+
+	parsed = parse_display_format(arg);
+	if (parsed == DISPLAY_FORMAT_UNKNOWN)
+		return error(_("unsupported output format '%s'"), arg);
+	*format = parsed;
+
+	return 0;
+}
+
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	const char *bundle_uri;
@@ -2102,6 +2136,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_CALLBACK(0, "output-format", &display_format, N_("format"), N_("output format"),
+			     opt_parse_output_format),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
@@ -2181,11 +2217,9 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		const char *format = "full";
 
 		git_config_get_string_tmp("fetch.output", &format);
-		if (!strcasecmp(format, "full"))
-			display_format = DISPLAY_FORMAT_FULL;
-		else if (!strcasecmp(format, "compact"))
-			display_format = DISPLAY_FORMAT_COMPACT;
-		else
+
+		display_format = parse_display_format(format);
+		if (display_format == DISPLAY_FORMAT_UNKNOWN)
 			die(_("invalid value for '%s': '%s'"),
 			    "fetch.output", format);
 	}
@@ -2339,7 +2373,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			max_children = fetch_parallel_config;
 
 		/* TODO should this also die if we have a previous partial-clone? */
-		result = fetch_multiple(&list, max_children);
+		result = fetch_multiple(&list, max_children, display_format);
 	}
 
 
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index b9dcdade63..662c960f94 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -24,14 +24,37 @@ test_expect_success 'fetch with invalid output format configuration' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch with invalid output format via command line' '
+	test_must_fail git fetch --output-format >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	error: option \`output-format${SQ} requires a value
+	EOF
+	test_cmp expect actual &&
+
+	test_must_fail git fetch --output-format= origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	error: unsupported output format ${SQ}${SQ}
+	EOF
+	test_cmp expect actual &&
+
+	test_must_fail git fetch --output-format=garbage origin >actual 2>&1 &&
+	cat >expect <<-EOF &&
+	error: unsupported output format ${SQ}garbage${SQ}
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
+	test_when_finished "rm -rf full-cfg full-cli" &&
+	git clone . full-cfg &&
+	git clone . full-cli &&
 	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
+
+	git -C full-cfg -c fetch.output=full fetch origin >actual-cfg 2>&1 &&
+	git -C full-cli fetch --output-format=full origin >actual-cli 2>&1 &&
+	test_cmp actual-cfg actual-cli &&
+
+	grep -e "->" actual-cfg | cut -c 22- >actual &&
 	cat >expect <<-\EOF &&
 	main                 -> origin/main
 	looooooooooooong-tag -> looooooooooooong-tag
@@ -40,13 +63,16 @@ test_expect_success 'fetch aligned output' '
 '
 
 test_expect_success 'fetch compact output' '
-	git clone . compact &&
+	test_when_finished "rm -rf compact-cfg compact-cli" &&
+	git clone . compact-cli &&
+	git clone . compact-cfg &&
 	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
+
+	git -C compact-cfg -c fetch.output=compact fetch origin >actual-cfg 2>&1 &&
+	git -C compact-cli fetch --output-format=compact origin >actual-cli 2>&1 &&
+	test_cmp actual-cfg actual-cli &&
+
+	grep -e "->" actual-cfg | cut -c 22- >actual &&
 	cat >expect <<-\EOF &&
 	main       -> origin/*
 	extraaa    -> *
@@ -54,6 +80,28 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch compact output with multiple remotes' '
+	test_when_finished "rm -rf compact-cfg compact-cli" &&
+
+	git clone . compact-cli &&
+	git -C compact-cli remote add second-remote "$PWD" &&
+	git clone . compact-cfg &&
+	git -C compact-cfg remote add second-remote "$PWD" &&
+	test_commit multi-commit &&
+
+	git -C compact-cfg -c fetch.output=compact fetch --all >actual-cfg 2>&1 &&
+	git -C compact-cli fetch --output-format=compact --all >actual-cli 2>&1 &&
+	test_cmp actual-cfg actual-cli &&
+
+	grep -e "->" actual-cfg | cut -c 22- >actual &&
+	cat >expect <<-\EOF &&
+	main         -> origin/*
+	multi-commit -> *
+	main       -> second-remote/*
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success 'fetch output with HEAD and --dry-run' '
 	test_when_finished "rm -rf head" &&
 	git clone . head &&
-- 
2.40.1
Junio C Hamano· Apr 27, 2023, 22:01 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

Patrick Steinhardt <ps@pks.im> writes:
Show 16 quoted lines
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 97a510649c..30099b2ac3 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -52,6 +52,13 @@ enum display_format {
>  	DISPLAY_FORMAT_UNKNOWN = 0,
>  	DISPLAY_FORMAT_FULL,
>  	DISPLAY_FORMAT_COMPACT,
> +	DISPLAY_FORMAT_MAX,
> +};
> +
> +static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
> +	NULL,
> +	"full",
> +	"compact",
>  };

Hmph, the _MAX thing that is only needed to size the array and never used elsewhere (i.e. parse_display_format() uses ARRAY_SIZE() of the thing, instead of the constant, and that is just fine) is an eyesore.

I wonder if
	static const char *const display_format[] = {
		[DISPLAY_FORMAT_UNKNOWN] = NULL,
		[DISPLAY_FORMAT_FULL] = "full",
		[DISPLAY_FORMAT_COMPACT] = "compact",
	};
would be easier to maintain?

I'll omit my usual "name your array in singular" lecture, as I think you've heard it already.

Thanks.
Glen Choo· Apr 28, 2023, 22:03 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

Junio C Hamano <gitster@pobox.com> writes:
Show 9 quoted lines
> I wonder if
>
> 	static const char *const display_format[] = {
> 		[DISPLAY_FORMAT_UNKNOWN] = NULL,
> 		[DISPLAY_FORMAT_FULL] = "full",
> 		[DISPLAY_FORMAT_COMPACT] = "compact",
> 	};
>
> would be easier to maintain?
It's easier to read, so I'd think so.
Patrick Steinhardt· May 3, 2023, 09:12 UTC · re: Glen Choo · lore

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

On Fri, Apr 28, 2023 at 03:03:43PM -0700, Glen Choo wrote:
Show 13 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
> 
> > I wonder if
> >
> > 	static const char *const display_format[] = {
> > 		[DISPLAY_FORMAT_UNKNOWN] = NULL,
> > 		[DISPLAY_FORMAT_FULL] = "full",
> > 		[DISPLAY_FORMAT_COMPACT] = "compact",
> > 	};
> >
> > would be easier to maintain?
> 
> It's easier to read, so I'd think so.
Yeah, I'll adopt this approach in v3.
Patrick
Glen Choo· Apr 28, 2023, 22:31 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

Patrick Steinhardt <ps@pks.im> writes:
Show 7 quoted lines
> @@ -1894,6 +1902,9 @@ static int fetch_multiple(struct string_list *list, int max_children)
>  		     "--no-write-commit-graph", NULL);
>  	add_options_to_argv(&argv);
>  
> +	if (format != DISPLAY_FORMAT_UNKNOWN)
> +		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
> +

I think these lines belong inside add_options_to_argv(), since that's also used to prepare argv for fetch_submodules(), so we'd also get support for --recurse-submodules. (I wish I had spotted that in v1, sorry. Thankfully they use the same helper function, so we only have to do this once.)

----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----
  diff --git a/builtin/fetch.c b/builtin/fetch.c
  index 422e29a914..7aa385aed5 100644
  --- a/builtin/fetch.c
  +++ b/builtin/fetch.c
  @@ -1796,8 +1796,11 @@ static int add_remote_or_group(const char *name, struct string_list *list)
    return 1;
  }
  -static void add_options_to_argv(struct strvec *argv)
  +static void add_options_to_argv(struct strvec *argv,
  +				enum display_format format)
  {
  /* Maybe this shouldn't be first, idk */
  +	if (format != DISPLAY_FORMAT_UNKNOWN)
  +		strvec_pushf(argv, "--output-format=%s", display_formats[format]);
    if (dry_run)
      strvec_push(argv, "--dry-run");
    if (prune != -1)
  @@ -1908,10 +1911,7 @@ static int fetch_multiple(struct string_list *list, int max_children,
    strvec_pushl(&argv, "-c", "fetch.bundleURI=",
          "fetch", "--append", "--no-auto-gc",
          "--no-write-commit-graph", NULL);
  -	add_options_to_argv(&argv);
  -
  -	if (format != DISPLAY_FORMAT_UNKNOWN)
  -		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
  +	add_options_to_argv(&argv, format);
    if (max_children != 1 && list->nr != 1) {
      struct parallel_fetch_state state = { argv.v, list, 0, 0 };
  @@ -2403,7 +2403,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
      if (max_children < 0)
        max_children = fetch_parallel_config;
  -		add_options_to_argv(&options);
  +		add_options_to_argv(&options, display_format);
      result = fetch_submodules(the_repository,
              &options,
              submodule_prefix,
----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----

I tested the result of that locally with --recurse-submodules, and it works.

Patrick Steinhardt· May 3, 2023, 09:43 UTC · re: Glen Choo · lore

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

On Fri, Apr 28, 2023 at 03:31:08PM -0700, Glen Choo wrote:
Show 61 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > @@ -1894,6 +1902,9 @@ static int fetch_multiple(struct string_list *list, int max_children)
> >  		     "--no-write-commit-graph", NULL);
> >  	add_options_to_argv(&argv);
> >  
> > +	if (format != DISPLAY_FORMAT_UNKNOWN)
> > +		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
> > +
> 
> I think these lines belong inside add_options_to_argv(), since that's
> also used to prepare argv for fetch_submodules(), so we'd also get
> support for --recurse-submodules. (I wish I had spotted that in v1,
> sorry. Thankfully they use the same helper function, so we only have to
> do this once.)
> 
> ----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----
>   diff --git a/builtin/fetch.c b/builtin/fetch.c
>   index 422e29a914..7aa385aed5 100644
>   --- a/builtin/fetch.c
>   +++ b/builtin/fetch.c
>   @@ -1796,8 +1796,11 @@ static int add_remote_or_group(const char *name, struct string_list *list)
>     return 1;
>   }
> 
>   -static void add_options_to_argv(struct strvec *argv)
>   +static void add_options_to_argv(struct strvec *argv,
>   +				enum display_format format)
>   {
>   /* Maybe this shouldn't be first, idk */
>   +	if (format != DISPLAY_FORMAT_UNKNOWN)
>   +		strvec_pushf(argv, "--output-format=%s", display_formats[format]);
>     if (dry_run)
>       strvec_push(argv, "--dry-run");
>     if (prune != -1)
>   @@ -1908,10 +1911,7 @@ static int fetch_multiple(struct string_list *list, int max_children,
>     strvec_pushl(&argv, "-c", "fetch.bundleURI=",
>           "fetch", "--append", "--no-auto-gc",
>           "--no-write-commit-graph", NULL);
>   -	add_options_to_argv(&argv);
>   -
>   -	if (format != DISPLAY_FORMAT_UNKNOWN)
>   -		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
>   +	add_options_to_argv(&argv, format);
> 
>     if (max_children != 1 && list->nr != 1) {
>       struct parallel_fetch_state state = { argv.v, list, 0, 0 };
>   @@ -2403,7 +2403,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
>       if (max_children < 0)
>         max_children = fetch_parallel_config;
> 
>   -		add_options_to_argv(&options);
>   +		add_options_to_argv(&options, display_format);
>       result = fetch_submodules(the_repository,
>               &options,
>               submodule_prefix,
> 
> ----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----
> 
> I tested the result of that locally with --recurse-submodules, and
> it works.

Unfortunately it doesn't quite work alright: while the porcelain format does indeed get inherited to the child process correctly, the parallel process API will cause us to group output per submodule-fetch. This has the consequence that stdout will be redirected into stderr, and that then breaks the assumption that all machine-parseable output goes to stdout.

My initial reflex is to just outright reject porcelain mode when submodule fetches are enabled. But that would require the caller to always explicitly pass `--recurse-submodules=off`, which isn't exactly great usability-wise.

The alternative would be to ungroup the output so that we can continue to print to the correct output streams. That works alright, and I've got a working version that does exactly that. But now we have the issue that the porcelain output is misleading: you cannot tell whether a specific reference update happens in the parent repository or in the submodule as that information is not part of the output.

I consider the second option to be much worse than the first option because it can cause scripts do to the wrong thing. So I'll send v3 with the first option, even though it's kind of an awful workaround. I'd be happy to hear any alternative proposals though.

Patrick
Patrick Steinhardt· May 3, 2023, 11:36 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 7/8] fetch: introduce new `--output-format` option

On Wed, May 03, 2023 at 11:43:31AM +0200, Patrick Steinhardt wrote:
Show 88 quoted lines
> On Fri, Apr 28, 2023 at 03:31:08PM -0700, Glen Choo wrote:
> > Patrick Steinhardt <ps@pks.im> writes:
> > 
> > > @@ -1894,6 +1902,9 @@ static int fetch_multiple(struct string_list *list, int max_children)
> > >  		     "--no-write-commit-graph", NULL);
> > >  	add_options_to_argv(&argv);
> > >  
> > > +	if (format != DISPLAY_FORMAT_UNKNOWN)
> > > +		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
> > > +
> > 
> > I think these lines belong inside add_options_to_argv(), since that's
> > also used to prepare argv for fetch_submodules(), so we'd also get
> > support for --recurse-submodules. (I wish I had spotted that in v1,
> > sorry. Thankfully they use the same helper function, so we only have to
> > do this once.)
> > 
> > ----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----
> >   diff --git a/builtin/fetch.c b/builtin/fetch.c
> >   index 422e29a914..7aa385aed5 100644
> >   --- a/builtin/fetch.c
> >   +++ b/builtin/fetch.c
> >   @@ -1796,8 +1796,11 @@ static int add_remote_or_group(const char *name, struct string_list *list)
> >     return 1;
> >   }
> > 
> >   -static void add_options_to_argv(struct strvec *argv)
> >   +static void add_options_to_argv(struct strvec *argv,
> >   +				enum display_format format)
> >   {
> >   /* Maybe this shouldn't be first, idk */
> >   +	if (format != DISPLAY_FORMAT_UNKNOWN)
> >   +		strvec_pushf(argv, "--output-format=%s", display_formats[format]);
> >     if (dry_run)
> >       strvec_push(argv, "--dry-run");
> >     if (prune != -1)
> >   @@ -1908,10 +1911,7 @@ static int fetch_multiple(struct string_list *list, int max_children,
> >     strvec_pushl(&argv, "-c", "fetch.bundleURI=",
> >           "fetch", "--append", "--no-auto-gc",
> >           "--no-write-commit-graph", NULL);
> >   -	add_options_to_argv(&argv);
> >   -
> >   -	if (format != DISPLAY_FORMAT_UNKNOWN)
> >   -		strvec_pushf(&argv, "--output-format=%s", display_formats[format]);
> >   +	add_options_to_argv(&argv, format);
> > 
> >     if (max_children != 1 && list->nr != 1) {
> >       struct parallel_fetch_state state = { argv.v, list, 0, 0 };
> >   @@ -2403,7 +2403,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
> >       if (max_children < 0)
> >         max_children = fetch_parallel_config;
> > 
> >   -		add_options_to_argv(&options);
> >   +		add_options_to_argv(&options, display_format);
> >       result = fetch_submodules(the_repository,
> >               &options,
> >               submodule_prefix,
> > 
> > ----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----
> > 
> > I tested the result of that locally with --recurse-submodules, and
> > it works.
> 
> Unfortunately it doesn't quite work alright: while the porcelain format
> does indeed get inherited to the child process correctly, the parallel
> process API will cause us to group output per submodule-fetch. This has
> the consequence that stdout will be redirected into stderr, and that
> then breaks the assumption that all machine-parseable output goes to
> stdout.
> 
> My initial reflex is to just outright reject porcelain mode when
> submodule fetches are enabled. But that would require the caller to
> always explicitly pass `--recurse-submodules=off`, which isn't exactly
> great usability-wise.
> 
> The alternative would be to ungroup the output so that we can continue
> to print to the correct output streams. That works alright, and I've got
> a working version that does exactly that. But now we have the issue that
> the porcelain output is misleading: you cannot tell whether a specific
> reference update happens in the parent repository or in the submodule as
> that information is not part of the output.
> 
> I consider the second option to be much worse than the first option
> because it can cause scripts do to the wrong thing. So I'll send v3 with
> the first option, even though it's kind of an awful workaround. I'd be
> happy to hear any alternative proposals though.
> 
> Patrick

I've gone with a slightly different variant of the first option that is inspired by `--negotiate-only`: instead of refusing to run, we disable submodule-fetches unless explicitly specified on the command line. In that case we return an error.

Patrick
Patrick Steinhardt· Apr 27, 2023, 11:13 UTC · re: Patrick Steinhardt · lore

[PATCH v2 8/8] fetch: introduce machine-parseable "porcelain" output format

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

``` <flag> <old-object-id> <new-object-id> <local-reference>\n ```

We assume two conditions which are generally true:
    - The old and new object IDs have fixed known widths and cannot
      contain spaces.
    - References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote. If usecases come up for this in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/config/fetch.txt  |  4 +-
 Documentation/fetch-options.txt |  4 +-
 Documentation/git-fetch.txt     | 17 ++++++-
 builtin/fetch.c                 | 47 +++++++++++++-----
 t/t5574-fetch-output.sh         | 84 +++++++++++++++++++++++++++++++++
 5 files changed, 139 insertions(+), 17 deletions(-)
Show changes to 5 files +139 −17

Documentation/config/fetch.txt, Documentation/fetch-options.txt, Documentation/git-fetch.txt, builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt
index 568f0f75b3..70734226c0 100644
--- a/Documentation/config/fetch.txt
+++ b/Documentation/config/fetch.txt
@@ -52,8 +52,8 @@ fetch.pruneTags::
 
 fetch.output::
 	Control how ref update status is printed. Valid values are
-	`full` and `compact`. Default value is `full`. See section
-	OUTPUT in linkgit:git-fetch[1] for detail.
+	`full`, `compact` and `porcelain`. Default value is `full`.
+	See section OUTPUT in linkgit:git-fetch[1] for detail.
 
 fetch.negotiationAlgorithm::
 	Control how information about the commits in the local repository
diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 654f96f79d..5ca8a67fe8 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -80,8 +80,8 @@ linkgit:git-config[1].
 
 --output-format::
 	Control how ref update status is printed. Valid values are
-	`full` and `compact`. Default value is `full`. See section
-	OUTPUT in linkgit:git-fetch[1] for detail.
+	`full`, `compact` and `porcelain`. Default value is `full`.
+	See section OUTPUT in linkgit:git-fetch[1] for detail.
 
 ifndef::git-pull[]
 --[no-]write-fetch-head::
diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt
index fba66f1460..efd22cd372 100644
--- a/Documentation/git-fetch.txt
+++ b/Documentation/git-fetch.txt
@@ -197,13 +197,26 @@ The output of "git fetch" depends on the transport method used; this
 section describes the output when fetching over the Git protocol
 (either locally or via ssh) and Smart HTTP protocol.
 
-The status of the fetch is output in tabular form, with each line
-representing the status of a single ref. Each line is of the form:
+The output format can be chosen either via the `fetch.output` config
+(see linkgit:git-config[1]), or via the `--output-format` switch.
+Supported values include:
+
+For the `full` and `compact` output formats, the status of the fetch is
+output in tabular, with each line representing the status of a single
+ref. Each line is of the form:
 
 -------------------------------
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+The `porcelain` output format is intended to be machine-parseable. In
+contrast to the human-readable output formats it thus prints to standard
+output instead of standard error. Each line is of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 30099b2ac3..abe6b8879d 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -52,6 +52,7 @@ enum display_format {
 	DISPLAY_FORMAT_UNKNOWN = 0,
 	DISPLAY_FORMAT_FULL,
 	DISPLAY_FORMAT_COMPACT,
+	DISPLAY_FORMAT_PORCELAIN,
 	DISPLAY_FORMAT_MAX,
 };
 
@@ -59,6 +60,7 @@ static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
 	NULL,
 	"full",
 	"compact",
+	"porcelain",
 };
 
 struct display_state {
@@ -752,8 +754,11 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 		break;
 	}
+	case DISPLAY_FORMAT_PORCELAIN:
+		/* We don't need to precompute anything here. */
+		break;
 	default:
-		BUG("unexpected display foramt %d", display_state->format);
+		BUG("unexpected display format %d", display_state->format);
 	}
 }
 
@@ -822,8 +827,12 @@ static void print_compact(struct display_state *display_state,
 static void display_ref_update(struct display_state *display_state, char code,
 			       const char *summary, const char *error,
 			       const char *remote, const char *local,
+			       const struct object_id *old_oid,
+			       const struct object_id *new_oid,
 			       int summary_width)
 {
+	FILE *f = stderr;
+
 	if (verbosity < 0)
 		return;
 
@@ -856,12 +865,17 @@ static void display_ref_update(struct display_state *display_state, char code,
 
 		break;
 	}
+	case DISPLAY_FORMAT_PORCELAIN:
+		strbuf_addf(&display_state->buf, "%c %s %s %s", code,
+			    oid_to_hex(old_oid), oid_to_hex(new_oid), local);
+		f = stdout;
+		break;
 	default:
 		BUG("unexpected display format %d", display_state->format);
 	};
 	strbuf_addch(&display_state->buf, '\n');
 
-	fputs(display_state->buf.buf, stderr);
+	fputs(display_state->buf.buf, f);
 }
 
 static int update_local_ref(struct ref *ref,
@@ -879,7 +893,8 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 		return 0;
 	}
 
@@ -892,7 +907,8 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 
@@ -903,12 +919,14 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return 1;
 		}
 	}
@@ -941,7 +959,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return r;
 	}
 
@@ -963,7 +982,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -975,12 +995,14 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 }
@@ -1221,7 +1243,9 @@ static int store_updated_refs(struct display_state *display_state,
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
 						   rm->name,
-						   "FETCH_HEAD", summary_width);
+						   "FETCH_HEAD",
+						   &rm->new_oid, &rm->old_oid,
+						   summary_width);
 			}
 		}
 	}
@@ -1361,6 +1385,7 @@ static int prune_refs(struct display_state *display_state,
 		for (ref = stale_refs; ref; ref = ref->next) {
 			display_ref_update(display_state, '-', _("[deleted]"), NULL,
 					   _("(none)"), ref->name,
+					   &ref->new_oid, &ref->old_oid,
 					   summary_width);
 			warn_dangling_symref(stderr, dangling_msg, ref->name);
 		}
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index 662c960f94..d88ad8af31 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -102,6 +102,72 @@ test_expect_success 'fetch compact output with multiple remotes' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch porcelain output' '
+	test_when_finished "rm -rf porcelain-cfg porcelain-cli" &&
+
+	# Set up a bunch of references that we can use to demonstrate different
+	# kinds of flag symbols in the output format.
+	MAIN_OLD=$(git rev-parse HEAD) &&
+	git branch "fast-forward" &&
+	git branch "deleted-branch" &&
+	git checkout -b force-updated &&
+	test_commit --no-tag force-update-old &&
+	FORCE_UPDATED_OLD=$(git rev-parse HEAD) &&
+	git checkout main &&
+
+	# Clone and pre-seed the repositories. We fetch references into two
+	# namespaces so that we can test that rejected and force-updated
+	# references are reported properly.
+	refspecs="refs/heads/*:refs/unforced/* +refs/heads/*:refs/forced/*" &&
+	git clone . porcelain-cli &&
+	git clone . porcelain-cfg &&
+	git -C porcelain-cfg fetch origin $refspecs &&
+	git -C porcelain-cli fetch origin $refspecs &&
+
+	# Now that we have set up the client repositories we can change our
+	# local references.
+	git branch new-branch &&
+	git branch -d deleted-branch &&
+	git checkout fast-forward &&
+	test_commit --no-tag fast-forward-new &&
+	FAST_FORWARD_NEW=$(git rev-parse HEAD) &&
+	git checkout force-updated &&
+	git reset --hard HEAD~ &&
+	test_commit --no-tag force-update-new &&
+	FORCE_UPDATED_NEW=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	- $MAIN_OLD $ZERO_OID refs/forced/deleted-branch
+	- $MAIN_OLD $ZERO_OID refs/unforced/deleted-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/unforced/fast-forward
+	! $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/unforced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/unforced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/forced/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/forced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/forced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/remotes/origin/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/remotes/origin/force-updated
+	* $ZERO_OID $MAIN_OLD refs/remotes/origin/new-branch
+	EOF
+
+	# Execute a dry-run fetch first. We do this to assert that the dry-run
+	# and non-dry-run fetches produces the same output. Execution of the
+	# fetch is expected to fail as we have a rejected reference update.
+	test_must_fail git -C porcelain-cfg -c fetch.output=porcelain fetch --dry-run --prune origin $refspecs >actual-dry-run-cfg &&
+	test_must_fail git -C porcelain-cli fetch --output-format=porcelain --dry-run --prune origin $refspecs >actual-dry-run-cli &&
+	test_cmp actual-dry-run-cfg actual-dry-run-cli &&
+	test_cmp expect actual-dry-run-cfg &&
+
+	# And now we perform a non-dry-run fetch.
+	test_must_fail git -C porcelain-cfg -c fetch.output=porcelain fetch --prune origin $refspecs >actual-cfg &&
+	test_must_fail git -C porcelain-cli fetch --output-format=porcelain --prune origin $refspecs >actual-cli &&
+	test_cmp actual-cfg actual-cli &&
+	test_cmp expect actual-cfg &&
+
+	# Ensure that the dry-run and non-dry-run output matches.
+	test_cmp actual-dry-run-cfg actual-cfg
+'
+
 test_expect_success 'fetch output with HEAD and --dry-run' '
 	test_when_finished "rm -rf head" &&
 	git clone . head &&
@@ -127,6 +193,24 @@ test_expect_success 'fetch output with HEAD and --dry-run' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+	COMMIT_ID=$(git rev-parse HEAD) &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID refs/heads/foo
+	EOF
+	test_cmp expect actual
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
-- 
2.40.1
Jacob Keller· Apr 27, 2023, 19:52 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 8/8] fetch: introduce machine-parseable "porcelain" output format

On 4/27/2023 4:13 AM, Patrick Steinhardt wrote:
Show 8 quoted lines
> A notable ommission here is that the output format does not include the
> remote from which a reference was fetched, which might be important
> information especially in the context of multi-remote fetches. But as
> such a format would require us to print the remote for every single
> reference update due to parallelizable fetches it feels wasteful for the
> most likely usecase, which is when fetching from a single remote. If
> usecases come up for this in the future though it is easy enough to add
> a new "porcelain-v2" format that adds this information.

Thanks for clarifying this. I think generally its ok to leave out since you can often infer what remote a ref came from (either because of refs/remotes, or because of how you configured your refspecs for fetching).

We do have the option to extend this in the future if needed, so I think that's ok. In that case if we wanted an even more verbose output I'd probably argue for something more structured like adding it as part of a JSON output.

Thanks, Jake

Glen Choo· Apr 28, 2023, 22:42 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2 8/8] fetch: introduce machine-parseable "porcelain" output format

Patrick Steinhardt <ps@pks.im> writes:
Show 8 quoted lines
> A notable ommission here is that the output format does not include the
> remote from which a reference was fetched, which might be important
> information especially in the context of multi-remote fetches. But as
> such a format would require us to print the remote for every single
> reference update due to parallelizable fetches it feels wasteful for the
> most likely usecase, which is when fetching from a single remote. If
> usecases come up for this in the future though it is easy enough to add
> a new "porcelain-v2" format that adds this information.

We discussed this elsewhere in the thread, but if we are just adding information (and not omitting existing information or shuffling it around), I would prefer for us to make the format extensible using flags than to add a whole new format enum. We can't imagine what other information users might want (maybe the remote ref name?), and it would be nice to avoid bumping the 'porcelain version' unnecessarily.

I agree that this format is good enough as a starting point, though.

It's not new to v2, but I've mentioned my reservations on --output-format on:

  https://lore.kernel.org/git/kl6ledo33ovx.fsf@chooglen-macbookpro.roam.corp.google.com/

I will be out of office all of next week. If, in the meantime, everyone else decides that --output-format=porcelain is good enough, I'm happy to accept the result.

Show 22 quoted lines
> @@ -127,6 +193,24 @@ test_expect_success 'fetch output with HEAD and --dry-run' '
>  	test_cmp expect actual
>  '
>  
> +test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
> +	test_when_finished "rm -rf head" &&
> +	git clone . head &&
> +	COMMIT_ID=$(git rev-parse HEAD) &&
> +
> +	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
> +	cat >expect <<-EOF &&
> +	* $ZERO_OID $COMMIT_ID FETCH_HEAD
> +	EOF
> +	test_cmp expect actual &&
> +
> +	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
> +	cat >expect <<-EOF &&
> +	* $ZERO_OID $COMMIT_ID refs/heads/foo
> +	EOF
> +	test_cmp expect actual
> +'
> +

Now that the earlier test also tests without --dry-run, shouldn't this one too?

Patrick Steinhardt· May 3, 2023, 11:34 UTC · re: Patrick Steinhardt · lore

[PATCH v3 0/8] fetch: introduce machine-parseable output

Hi,

this is the third version of my patch series to introduce a machine-parseable output for git-fetch(1).

Changes compared to v2:
    - A new commit that fixes `--no-recurse-submodules` not being
      honored for multi-remote fetches was added. This fixes issues that
      would otherwise be hit later in the patch series.
    - Tests were adjusted to explicitly verify stdout and stderr as
      standalone units instead of redirecting 2>&1.
    - Instead of introducing `--output-format` and a new "porcelain"
      format I've changed this to instead introduce `--porcelain`. What
      finally convinced me to go this way is Glen's list of preexisting
      commands that have `--porcelain`. So I decided to not be the only
      outlier here.
    - The DISPLAY_FORMAT_MAX macro was dropped in favor of Junio's
      proposal that uses designated initializers for the array.
    - Similar to `--negotiate-only`, the porcelain output is now
      incompatible with `--recurse-submodules=[yes|on-demand]`. This is
      done to avoid ambiguity when references in both the superproject
      and any of its submodules is updated.
Thanks again for all the feedback!
Patrick
Patrick Steinhardt (8):
  fetch: fix `--no-recurse-submodules` with multi-remote fetches
  fetch: split out tests for output format
  fetch: add a test to exercise invalid output formats
  fetch: fix missing from-reference when fetching HEAD:foo
  fetch: introduce `display_format` enum
  fetch: move display format parsing into main function
  fetch: move option related variables into main function
  fetch: introduce machine-parseable "porcelain" output format
 Documentation/fetch-options.txt |   6 +
 Documentation/git-fetch.txt     |   9 +
 builtin/fetch.c                 | 437 +++++++++++++++++++-------------
 t/t5510-fetch.sh                |  53 ----
 t/t5526-fetch-submodules.sh     |  31 +++
 t/t5574-fetch-output.sh         | 231 +++++++++++++++++
 6 files changed, 543 insertions(+), 224 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Range-diff against v2:
-:  ---------- > 1:  4b2b0cfe15 fetch: fix `--no-recurse-submodules` with multi-remote fetches
1:  0d0d50d14c = 2:  6ebc7450ba fetch: split out tests for output format
2:  29d2c58914 ! 3:  78479922ac fetch: add a test to exercise invalid output formats
    @@ t/t5574-fetch-output.sh: export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
     +	test_when_finished "rm -rf clone" &&
     +	git clone . clone &&
     +
    -+	test_must_fail git -C clone -c fetch.output= fetch origin >actual 2>&1 &&
    ++	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
     +	EOF
    -+	test_cmp expect actual &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err &&
     +
    -+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual 2>&1 &&
    ++	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
     +	EOF
    -+	test_cmp expect actual
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err
     +'
     +
      test_expect_success 'fetch aligned output' '
3:  d1fb6eeae7 ! 4:  46e1266ab0 fetch: fix missing from-reference when fetching HEAD:foo
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
      	test_cmp expect actual
      '
      
    -+test_expect_success 'fetch output with HEAD and --dry-run' '
    ++test_expect_success 'fetch output with HEAD' '
     +	test_when_finished "rm -rf head" &&
     +	git clone . head &&
     +
    -+	git -C head fetch --dry-run origin HEAD >actual 2>&1 &&
    ++	git -C head fetch --dry-run origin HEAD >actual.out 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	From $(test-tool path-utils real_path .)/.
     +	 * branch            HEAD       -> FETCH_HEAD
     +	EOF
    -+	test_cmp expect actual &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err &&
     +
    -+	git -C head fetch origin HEAD >actual 2>&1 &&
    -+	test_cmp expect actual &&
    ++	git -C head fetch origin HEAD >actual.out 2>actual.err &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err &&
     +
    -+	git -C head fetch --dry-run origin HEAD:foo >actual 2>&1 &&
    ++	git -C head fetch --dry-run origin HEAD:foo >actual.out 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	From $(test-tool path-utils real_path .)/.
     +	 * [new ref]         HEAD       -> foo
     +	EOF
    -+	test_cmp expect actual &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err &&
     +
    -+	git -C head fetch origin HEAD:foo >actual 2>&1 &&
    -+	test_cmp expect actual
    ++	git -C head fetch origin HEAD:foo >actual.out 2>actual.err &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err
     +'
     +
      test_expect_success '--no-show-forced-updates' '
4:  b545bf8bb9 ! 5:  acc0f7f520 fetch: introduce `display_format` enum
    @@ builtin/fetch.c: static void display_state_init(struct display_state *display_st
     +		break;
     +	}
     +	default:
    -+		BUG("unexpected display foramt %d", display_state->format);
    ++		BUG("unexpected display format %d", display_state->format);
      	}
      }
      
5:  4990d35998 = 6:  cd23440128 fetch: move display format parsing into main function
6:  cfe84129ab = 7:  edbc31013f fetch: move option related variables into main function
7:  0335e5eeb4 < -:  ---------- fetch: introduce new `--output-format` option
8:  d7c1bc1a80 ! 8:  e132d9494e fetch: introduce machine-parseable "porcelain" output format
    @@ Commit message
         information especially in the context of multi-remote fetches. But as
         such a format would require us to print the remote for every single
         reference update due to parallelizable fetches it feels wasteful for the
    -    most likely usecase, which is when fetching from a single remote. If
    -    usecases come up for this in the future though it is easy enough to add
    -    a new "porcelain-v2" format that adds this information.
    +    most likely usecase, which is when fetching from a single remote.
    +
    +    In a similar spirit, a second restriction is that this cannot be used
    +    with `--recurse-submodules`. This is because any reference updates would
    +    be ambiguous without also printing the repository in which the update
    +    happens.
    +
    +    Considering that both multi-remote and submodule fetches are rather
    +    niche and likely not going to be useful for the majority of usecases
    +    these omissions feel acceptable. If usecases for either of these come up
    +    in the future though it is easy enough to add a new "porcelain-v2"
    +    format that adds this information.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    - ## Documentation/config/fetch.txt ##
    -@@ Documentation/config/fetch.txt: fetch.pruneTags::
    - 
    - fetch.output::
    - 	Control how ref update status is printed. Valid values are
    --	`full` and `compact`. Default value is `full`. See section
    --	OUTPUT in linkgit:git-fetch[1] for detail.
    -+	`full`, `compact` and `porcelain`. Default value is `full`.
    -+	See section OUTPUT in linkgit:git-fetch[1] for detail.
    - 
    - fetch.negotiationAlgorithm::
    - 	Control how information about the commits in the local repository
    -
      ## Documentation/fetch-options.txt ##
     @@ Documentation/fetch-options.txt: linkgit:git-config[1].
    + --dry-run::
    + 	Show what would be done, without making any changes.
      
    - --output-format::
    - 	Control how ref update status is printed. Valid values are
    --	`full` and `compact`. Default value is `full`. See section
    --	OUTPUT in linkgit:git-fetch[1] for detail.
    -+	`full`, `compact` and `porcelain`. Default value is `full`.
    -+	See section OUTPUT in linkgit:git-fetch[1] for detail.
    - 
    ++--porcelain::
    ++	Print the output to standard output in an easy-to-parse format for
    ++	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
    +++
    ++This is incompatible with `--recurse-submodules=[yes|on-demand]`.
    ++
      ifndef::git-pull[]
      --[no-]write-fetch-head::
    + 	Write the list of remote refs fetched in the `FETCH_HEAD`
     
      ## Documentation/git-fetch.txt ##
    -@@ Documentation/git-fetch.txt: The output of "git fetch" depends on the transport method used; this
    - section describes the output when fetching over the Git protocol
    - (either locally or via ssh) and Smart HTTP protocol.
    - 
    --The status of the fetch is output in tabular form, with each line
    --representing the status of a single ref. Each line is of the form:
    -+The output format can be chosen either via the `fetch.output` config
    -+(see linkgit:git-config[1]), or via the `--output-format` switch.
    -+Supported values include:
    -+
    -+For the `full` and `compact` output formats, the status of the fetch is
    -+output in tabular, with each line representing the status of a single
    -+ref. Each line is of the form:
    - 
    - -------------------------------
    +@@ Documentation/git-fetch.txt: representing the status of a single ref. Each line is of the form:
       <flag> <summary> <from> -> <to> [<reason>]
      -------------------------------
      
    -+The `porcelain` output format is intended to be machine-parseable. In
    -+contrast to the human-readable output formats it thus prints to standard
    -+output instead of standard error. Each line is of the form:
    ++When using `--porcelain`, the output format is intended to be
    ++machine-parseable. In contrast to the human-readable output formats it
    ++thus prints to standard output instead of standard error. Each line is
    ++of the form:
     +
     +-------------------------------
     +<flag> <old-object-id> <new-object-id> <local-reference>
    @@ builtin/fetch.c: enum display_format {
      	DISPLAY_FORMAT_FULL,
      	DISPLAY_FORMAT_COMPACT,
     +	DISPLAY_FORMAT_PORCELAIN,
    - 	DISPLAY_FORMAT_MAX,
    - };
    - 
    -@@ builtin/fetch.c: static const char * const display_formats[DISPLAY_FORMAT_MAX] = {
    - 	NULL,
    - 	"full",
    - 	"compact",
    -+	"porcelain",
      };
      
      struct display_state {
    @@ builtin/fetch.c: static void display_state_init(struct display_state *display_st
     +		/* We don't need to precompute anything here. */
     +		break;
      	default:
    --		BUG("unexpected display foramt %d", display_state->format);
    -+		BUG("unexpected display format %d", display_state->format);
    + 		BUG("unexpected display format %d", display_state->format);
      	}
    - }
    - 
     @@ builtin/fetch.c: static void print_compact(struct display_state *display_state,
      static void display_ref_update(struct display_state *display_state, char code,
      			       const char *summary, const char *error,
    @@ builtin/fetch.c: static int prune_refs(struct display_state *display_state,
      					   summary_width);
      			warn_dangling_symref(stderr, dangling_msg, ref->name);
      		}
    +@@ builtin/fetch.c: static int add_remote_or_group(const char *name, struct string_list *list)
    + 	return 1;
    + }
    + 
    +-static void add_options_to_argv(struct strvec *argv)
    ++static void add_options_to_argv(struct strvec *argv,
    ++				enum display_format format)
    + {
    + 	if (dry_run)
    + 		strvec_push(argv, "--dry-run");
    +@@ builtin/fetch.c: static void add_options_to_argv(struct strvec *argv)
    + 		strvec_push(argv, "--ipv6");
    + 	if (!write_fetch_head)
    + 		strvec_push(argv, "--no-write-fetch-head");
    ++	if (format == DISPLAY_FORMAT_PORCELAIN)
    ++		strvec_pushf(argv, "--porcelain");
    + }
    + 
    + /* Fetch multiple remotes in parallel */
    +@@ builtin/fetch.c: struct parallel_fetch_state {
    + 	const char **argv;
    + 	struct string_list *remotes;
    + 	int next, result;
    ++	enum display_format format;
    + };
    + 
    + static int fetch_next_remote(struct child_process *cp,
    +@@ builtin/fetch.c: static int fetch_next_remote(struct child_process *cp,
    + 	strvec_push(&cp->args, remote);
    + 	cp->git_cmd = 1;
    + 
    +-	if (verbosity >= 0)
    ++	if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)
    + 		printf(_("Fetching %s\n"), remote);
    + 
    + 	return 1;
    +@@ builtin/fetch.c: static int fetch_finished(int result, struct strbuf *out,
    + 	return 0;
    + }
    + 
    +-static int fetch_multiple(struct string_list *list, int max_children)
    ++static int fetch_multiple(struct string_list *list, int max_children,
    ++			  enum display_format format)
    + {
    + 	int i, result = 0;
    + 	struct strvec argv = STRVEC_INIT;
    +@@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_children)
    + 
    + 	strvec_pushl(&argv, "fetch", "--append", "--no-auto-gc",
    + 		     "--no-write-commit-graph", NULL);
    +-	add_options_to_argv(&argv);
    ++	add_options_to_argv(&argv, format);
    + 
    + 	if (max_children != 1 && list->nr != 1) {
    +-		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
    ++		struct parallel_fetch_state state = { argv.v, list, 0, 0, format };
    + 		const struct run_process_parallel_opts opts = {
    + 			.tr2_category = "fetch",
    + 			.tr2_label = "parallel/fetch",
    +@@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_children)
    + 
    + 			strvec_pushv(&cmd.args, argv.v);
    + 			strvec_push(&cmd.args, name);
    +-			if (verbosity >= 0)
    ++			if (verbosity >= 0 && format != DISPLAY_FORMAT_PORCELAIN)
    + 				printf(_("Fetching %s\n"), name);
    + 			cmd.git_cmd = 1;
    + 			if (run_command(&cmd)) {
    +@@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv,
    + 	return exit_code;
    + }
    + 
    ++static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
    ++{
    ++	enum display_format *format = opt->value;
    ++	*format = DISPLAY_FORMAT_PORCELAIN;
    ++	return 0;
    ++}
    ++
    + int cmd_fetch(int argc, const char **argv, const char *prefix)
    + {
    + 	const char *bundle_uri;
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
    + 		OPT_BOOL(0, "dry-run", &dry_run,
    + 			 N_("dry run")),
    ++		OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
    ++			       PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
    + 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
    + 			 N_("write fetched references to the FETCH_HEAD file")),
    + 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 		fetch_config_from_gitmodules(sfjc, rs);
    + 	}
    + 
    ++	if (display_format == DISPLAY_FORMAT_PORCELAIN) {
    ++		switch (recurse_submodules_cli) {
    ++		case RECURSE_SUBMODULES_OFF:
    ++		case RECURSE_SUBMODULES_DEFAULT:
    ++			/*
    ++			 * Reference updates in submodules would be ambiguous
    ++			 * in porcelain mode, so we reject this combination.
    ++			 */
    ++			recurse_submodules = RECURSE_SUBMODULES_OFF;
    ++			break;
    ++
    ++		default:
    ++			die(_("options '%s' and '%s' cannot be used together"),
    ++			    "--porcelain", "--recurse-submodules");
    ++		}
    ++	}
    ++
    + 	if (negotiate_only && !negotiation_tip.nr)
    + 		die(_("--negotiate-only needs one or more --negotiation-tip=*"));
    + 
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 			max_children = fetch_parallel_config;
    + 
    + 		/* TODO should this also die if we have a previous partial-clone? */
    +-		result = fetch_multiple(&list, max_children);
    ++		result = fetch_multiple(&list, max_children, display_format);
    + 	}
    + 
    + 
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 		if (max_children < 0)
    + 			max_children = fetch_parallel_config;
    + 
    +-		add_options_to_argv(&options);
    ++		add_options_to_argv(&options, display_format);
    + 		result = fetch_submodules(the_repository,
    + 					  &options,
    + 					  submodule_prefix,
     
      ## t/t5574-fetch-output.sh ##
    -@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output with multiple remotes' '
    +@@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
      	test_cmp expect actual
      '
      
     +test_expect_success 'fetch porcelain output' '
    -+	test_when_finished "rm -rf porcelain-cfg porcelain-cli" &&
    ++	test_when_finished "rm -rf porcelain" &&
     +
     +	# Set up a bunch of references that we can use to demonstrate different
     +	# kinds of flag symbols in the output format.
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output with multiple
     +	# namespaces so that we can test that rejected and force-updated
     +	# references are reported properly.
     +	refspecs="refs/heads/*:refs/unforced/* +refs/heads/*:refs/forced/*" &&
    -+	git clone . porcelain-cli &&
    -+	git clone . porcelain-cfg &&
    -+	git -C porcelain-cfg fetch origin $refspecs &&
    -+	git -C porcelain-cli fetch origin $refspecs &&
    ++	git clone . porcelain &&
    ++	git -C porcelain fetch origin $refspecs &&
     +
     +	# Now that we have set up the client repositories we can change our
     +	# local references.
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output with multiple
     +	# Execute a dry-run fetch first. We do this to assert that the dry-run
     +	# and non-dry-run fetches produces the same output. Execution of the
     +	# fetch is expected to fail as we have a rejected reference update.
    -+	test_must_fail git -C porcelain-cfg -c fetch.output=porcelain fetch --dry-run --prune origin $refspecs >actual-dry-run-cfg &&
    -+	test_must_fail git -C porcelain-cli fetch --output-format=porcelain --dry-run --prune origin $refspecs >actual-dry-run-cli &&
    -+	test_cmp actual-dry-run-cfg actual-dry-run-cli &&
    -+	test_cmp expect actual-dry-run-cfg &&
    ++	test_must_fail git -C porcelain fetch \
    ++		--porcelain --dry-run --prune origin $refspecs >actual &&
    ++	test_cmp expect actual &&
     +
     +	# And now we perform a non-dry-run fetch.
    -+	test_must_fail git -C porcelain-cfg -c fetch.output=porcelain fetch --prune origin $refspecs >actual-cfg &&
    -+	test_must_fail git -C porcelain-cli fetch --output-format=porcelain --prune origin $refspecs >actual-cli &&
    -+	test_cmp actual-cfg actual-cli &&
    -+	test_cmp expect actual-cfg &&
    -+
    -+	# Ensure that the dry-run and non-dry-run output matches.
    -+	test_cmp actual-dry-run-cfg actual-cfg
    ++	test_must_fail git -C porcelain fetch \
    ++		--porcelain --prune origin $refspecs >actual 2>stderr &&
    ++	test_cmp expect actual &&
    ++	test_must_be_empty stderr
     +'
     +
    - test_expect_success 'fetch output with HEAD and --dry-run' '
    ++test_expect_success 'fetch porcelain with multiple remotes' '
    ++	test_when_finished "rm -rf porcelain" &&
    ++
    ++	git clone . porcelain &&
    ++	git -C porcelain remote add second-remote "$PWD" &&
    ++	git -C porcelain fetch second-remote &&
    ++
    ++	test_commit --no-tag multi-commit &&
    ++	old_commit=$(git rev-parse HEAD~) &&
    ++	new_commit=$(git rev-parse HEAD) &&
    ++
    ++	cat >expect <<-EOF &&
    ++	  $old_commit $new_commit refs/remotes/origin/force-updated
    ++	  $old_commit $new_commit refs/remotes/second-remote/force-updated
    ++	EOF
    ++
    ++	git -C porcelain fetch --porcelain --all >actual 2>stderr &&
    ++	test_cmp expect actual &&
    ++	test_must_be_empty stderr
    ++'
    ++
    ++test_expect_success 'fetch porcelain refuses to work with submodules' '
    ++	test_when_finished "rm -rf porcelain" &&
    ++
    ++	cat >expect <<-EOF &&
    ++	fatal: options ${SQ}--porcelain${SQ} and ${SQ}--recurse-submodules${SQ} cannot be used together
    ++	EOF
    ++
    ++	git init porcelain &&
    ++	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=yes 2>stderr &&
    ++	test_cmp expect stderr &&
    ++
    ++	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=on-demand 2>stderr &&
    ++	test_cmp expect stderr
    ++'
    ++
    + test_expect_success 'fetch output with HEAD' '
      	test_when_finished "rm -rf head" &&
      	git clone . head &&
    -@@ t/t5574-fetch-output.sh: test_expect_success 'fetch output with HEAD and --dry-run' '
    - 	test_cmp expect actual
    +@@ t/t5574-fetch-output.sh: test_expect_success 'fetch output with HEAD' '
    + 	test_cmp expect actual.err
      '
      
    -+test_expect_success 'fetch porcelain output with HEAD and --dry-run' '
    ++test_expect_success 'fetch porcelain output with HEAD' '
     +	test_when_finished "rm -rf head" &&
     +	git clone . head &&
     +	COMMIT_ID=$(git rev-parse HEAD) &&
     +
    -+	git -C head fetch --output-format=porcelain --dry-run origin HEAD >actual &&
    ++	git -C head fetch --porcelain --dry-run origin HEAD >actual &&
     +	cat >expect <<-EOF &&
     +	* $ZERO_OID $COMMIT_ID FETCH_HEAD
     +	EOF
     +	test_cmp expect actual &&
     +
    -+	git -C head fetch --output-format=porcelain --dry-run origin HEAD:foo >actual &&
    ++	git -C head fetch --porcelain origin HEAD >actual &&
    ++	test_cmp expect actual &&
    ++
    ++	git -C head fetch --porcelain --dry-run origin HEAD:foo >actual &&
     +	cat >expect <<-EOF &&
     +	* $ZERO_OID $COMMIT_ID refs/heads/foo
     +	EOF
    ++	test_cmp expect actual &&
    ++
    ++	git -C head fetch --porcelain origin HEAD:foo >actual &&
     +	test_cmp expect actual
     +'
     +
-- 
2.40.1
Patrick Steinhardt· May 3, 2023, 11:34 UTC · re: Patrick Steinhardt · lore

[PATCH v3 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

When running `git fetch --no-recurse-submodules`, the exectation is that we don't fetch any submodules. And while this works for fetches of a single remote, it doesn't when fetching multiple remotes at once. The result is that we do recurse into submodules even though the user has explicitly asked us not to.

This is because while we pass on `--recurse-submodules={yes,on-demand}` if specified by the user, we don't pass on `--no-recurse-submodules` to the subprocess spawned to perform the submodule fetch.

Fix this by also forwarding this flag as expected.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c             |  2 ++
 t/t5526-fetch-submodules.sh | 31 +++++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)
Show changes to 2 files +33 −0

builtin/fetch.c, t/t5526-fetch-submodules.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index c310d89878..08d7fc7233 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1876,6 +1876,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--keep");
 	if (recurse_submodules == RECURSE_SUBMODULES_ON)
 		strvec_push(argv, "--recurse-submodules");
+	else if (recurse_submodules == RECURSE_SUBMODULES_OFF)
+		strvec_push(argv, "--no-recurse-submodules");
 	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
 		strvec_push(argv, "--recurse-submodules=on-demand");
 	if (tags == TAGS_SET)
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index dcdbe26a08..162e5bac2f 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -1180,4 +1180,35 @@ test_expect_success 'fetch --all with --recurse-submodules with multiple' '
 	test_line_count = 2 fetch-subs
 '
 
+test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
+	test_when_finished "git -C downstream remote remove second" &&
+
+	# We need to add a second remote, otherwise --all falls back to the
+	# normal fetch-one case.
+	git -C downstream remote add second .. &&
+	git -C downstream fetch --all &&
+
+	add_submodule_commits &&
+	add_superproject_commits &&
+	old_commit=$(git rev-parse --short HEAD~) &&
+	new_commit=$(git rev-parse --short HEAD) &&
+
+	git -C downstream fetch --all --no-recurse-submodules >actual.out 2>actual.err &&
+
+	cat >expect.out <<-EOF &&
+	Fetching origin
+	Fetching second
+	EOF
+
+	cat >expect.err <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	   $old_commit..$new_commit  super      -> origin/super
+	From ..
+	   $old_commit..$new_commit  super      -> second/super
+	EOF
+
+	test_cmp expect.out actual.out &&
+	test_cmp expect.err actual.err
+'
+
 test_done
-- 
2.40.1
Glen Choo· May 8, 2023, 22:51 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v3 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

Patrick Steinhardt <ps@pks.im> writes:
Show 31 quoted lines
> When running `git fetch --no-recurse-submodules`, the exectation is that
> we don't fetch any submodules. And while this works for fetches of a
> single remote, it doesn't when fetching multiple remotes at once. The
> result is that we do recurse into submodules even though the user has
> explicitly asked us not to.
>
> This is because while we pass on `--recurse-submodules={yes,on-demand}`
> if specified by the user, we don't pass on `--no-recurse-submodules` to
> the subprocess spawned to perform the submodule fetch.
>
> Fix this by also forwarding this flag as expected.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  builtin/fetch.c             |  2 ++
>  t/t5526-fetch-submodules.sh | 31 +++++++++++++++++++++++++++++++
>  2 files changed, 33 insertions(+)
>
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index c310d89878..08d7fc7233 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -1876,6 +1876,8 @@ static void add_options_to_argv(struct strvec *argv)
>  		strvec_push(argv, "--keep");
>  	if (recurse_submodules == RECURSE_SUBMODULES_ON)
>  		strvec_push(argv, "--recurse-submodules");
> +	else if (recurse_submodules == RECURSE_SUBMODULES_OFF)
> +		strvec_push(argv, "--no-recurse-submodules");
>  	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
>  		strvec_push(argv, "--recurse-submodules=on-demand");
>  	if (tags == TAGS_SET)
Makes sense.

I wondered for a bit whether this should have been checking recurse_submodules_cli (the actual CLI flag) back in 386c076a86 (fetch --negotiate-only: do not update submodules, 2022-01-18). I think this current state is correct, though. After we have told the superproject what submodule recursion mode to use, we want to continue using that mode when recursing through submodules regardless of whether that mode originally came from the CLI or config.

Show 30 quoted lines
> +test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
> +	test_when_finished "git -C downstream remote remove second" &&
> +
> +	# We need to add a second remote, otherwise --all falls back to the
> +	# normal fetch-one case.
> +	git -C downstream remote add second .. &&
> +	git -C downstream fetch --all &&
> +
> +	add_submodule_commits &&
> +	add_superproject_commits &&
> +	old_commit=$(git rev-parse --short HEAD~) &&
> +	new_commit=$(git rev-parse --short HEAD) &&
> +
> +	git -C downstream fetch --all --no-recurse-submodules >actual.out 2>actual.err &&
> +
> +	cat >expect.out <<-EOF &&
> +	Fetching origin
> +	Fetching second
> +	EOF
> +
> +	cat >expect.err <<-EOF &&
> +	From $(test-tool path-utils real_path .)/.
> +	   $old_commit..$new_commit  super      -> origin/super
> +	From ..
> +	   $old_commit..$new_commit  super      -> second/super
> +	EOF
> +
> +	test_cmp expect.out actual.out &&
> +	test_cmp expect.err actual.err
> +'

The test looks okay, though is there a reason you didn't copy the style of the previous test? It is nearly exactly what you want, I think, like (untested)

  test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
    test_when_finished "rm -fr src_clone" &&
    git clone --recurse-submodules src src_clone &&
    (
      cd src_clone &&
      git remote add secondary ../src &&
      git config submodule.recurse true &&
      git config fetch.parallel 0 &&
      git fetch --all 2>../fetch-log
    ) &&
    grep "Fetching submodule" fetch-log >fetch-subs &&
    test_must_be_empty fetch-subs
  '
which has the handy benefit of not needing the test-tools invocation.
Patrick Steinhardt· May 9, 2023, 12:41 UTC · re: Glen Choo · lore

Re: [PATCH v3 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

On Mon, May 08, 2023 at 03:51:52PM -0700, Glen Choo wrote:
> Patrick Steinhardt <ps@pks.im> writes:
[snip]
Show 50 quoted lines
> > +test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
> > +	test_when_finished "git -C downstream remote remove second" &&
> > +
> > +	# We need to add a second remote, otherwise --all falls back to the
> > +	# normal fetch-one case.
> > +	git -C downstream remote add second .. &&
> > +	git -C downstream fetch --all &&
> > +
> > +	add_submodule_commits &&
> > +	add_superproject_commits &&
> > +	old_commit=$(git rev-parse --short HEAD~) &&
> > +	new_commit=$(git rev-parse --short HEAD) &&
> > +
> > +	git -C downstream fetch --all --no-recurse-submodules >actual.out 2>actual.err &&
> > +
> > +	cat >expect.out <<-EOF &&
> > +	Fetching origin
> > +	Fetching second
> > +	EOF
> > +
> > +	cat >expect.err <<-EOF &&
> > +	From $(test-tool path-utils real_path .)/.
> > +	   $old_commit..$new_commit  super      -> origin/super
> > +	From ..
> > +	   $old_commit..$new_commit  super      -> second/super
> > +	EOF
> > +
> > +	test_cmp expect.out actual.out &&
> > +	test_cmp expect.err actual.err
> > +'
> 
> The test looks okay, though is there a reason you didn't copy the style
> of the previous test? It is nearly exactly what you want, I think, like
> (untested)
> 
>   test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
>     test_when_finished "rm -fr src_clone" &&
>     git clone --recurse-submodules src src_clone &&
>     (
>       cd src_clone &&
>       git remote add secondary ../src &&
>       git config submodule.recurse true &&
>       git config fetch.parallel 0 &&
>       git fetch --all 2>../fetch-log
>     ) &&
>     grep "Fetching submodule" fetch-log >fetch-subs &&
>     test_must_be_empty fetch-subs
>   '
> 
> which has the handy benefit of not needing the test-tools invocation.
That is indeed much simpler, thanks.
Patrick
Patrick Steinhardt· May 3, 2023, 11:34 UTC · re: Patrick Steinhardt · lore

[PATCH v3 2/8] fetch: split out tests for output format

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Show changes to 2 files +63 −53

t/t5510-fetch.sh, t/t5574-fetch-output.sh

diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index dc44da9c79..4f289063ce 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
new file mode 100755
index 0000000000..f91b654d38
--- /dev/null
+++ b/t/t5574-fetch-output.sh
@@ -0,0 +1,63 @@
+#!/bin/sh
+
+test_description='git fetch output format'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+
+test_expect_success 'fetch aligned output' '
+	git clone . full-output &&
+	test_commit looooooooooooong-tag &&
+	(
+		cd full-output &&
+		git -c fetch.output=full fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main                 -> origin/main
+	looooooooooooong-tag -> looooooooooooong-tag
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'fetch compact output' '
+	git clone . compact &&
+	test_commit extraaa &&
+	(
+		cd compact &&
+		git -c fetch.output=compact fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main       -> origin/*
+	extraaa    -> *
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success '--no-show-forced-updates' '
+	mkdir forced-updates &&
+	(
+		cd forced-updates &&
+		git init &&
+		test_commit 1 &&
+		test_commit 2
+	) &&
+	git clone forced-updates forced-update-clone &&
+	git clone forced-updates no-forced-update-clone &&
+	git -C forced-updates reset --hard HEAD~1 &&
+	(
+		cd forced-update-clone &&
+		git fetch --show-forced-updates origin 2>output &&
+		test_i18ngrep "(forced update)" output
+	) &&
+	(
+		cd no-forced-update-clone &&
+		git fetch --no-show-forced-updates origin 2>output &&
+		test_i18ngrep ! "(forced update)" output
+	)
+'
+
+test_done
-- 
2.40.1
Patrick Steinhardt· May 3, 2023, 11:34 UTC · re: Patrick Steinhardt · lore

[PATCH v3 3/8] fetch: add a test to exercise invalid output formats

Add a testcase that exercises the logic when an invalid output format is passed via the `fetch.output` configuration.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5574-fetch-output.sh | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
Show changes to t/t5574-fetch-output.sh +19 −0
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index f91b654d38..a09750d225 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -7,6 +7,25 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
-- 
2.40.1
Patrick Steinhardt· May 3, 2023, 11:34 UTC · re: Patrick Steinhardt · lore

[PATCH v3 4/8] fetch: fix missing from-reference when fetching HEAD:foo

`store_updated_refs()` parses the remote reference for two purposes:
    - It gets used as a note when writing FETCH_HEAD.
    - It is passed through to `display_ref_update()` to display
      updated references in the following format:
      ```
       * branch               master          -> master
      ```

In most cases, the parsed remote reference is the prettified reference name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case. But it is not intended when displaying the updated references and would cause us to miss the left-hand side of the displayed reference update:

```
$ git fetch origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands show the left-hand side as expected:

```
$ git fetch origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD
$ git fetch origin master
From https://github.com/git/git
 * branch                  master     -> FETCH_HEAD
 * branch                  master     -> origin/master
```

The logic of how we compute the remote reference name that we ultimately pass to `display_ref_update()` is not easy to follow. There are three different cases here:

    - When the remote reference name is "HEAD" we set the remote
      reference name to the empty string. This is the case that causes
      the bug to occur, where we would indeed want to print "HEAD"
      instead of the empty string. This is what `prettify_refname()`
      would return.
    - When the remote reference name has a well-known prefix then we
      strip this prefix. This matches what `prettify_refname()` does.
    - Otherwise, we keep the fully qualified reference name. This also
      matches what `prettify_refname()` does.

As the return value of `prettify_refname()` would do the correct thing for us in all three cases, we can fix the bug by passing through the full remote reference name to `display_ref_update()`, which learns to call `prettify_refname()`. At the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c         | 37 +++++++++++++++++++------------------
 t/t5574-fetch-output.sh | 29 +++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+), 18 deletions(-)
Show changes to 2 files +48 −18

builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 08d7fc7233..6aecf549e8 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index a09750d225..6e0f7e0046 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -56,6 +56,35 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD:foo >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
-- 
2.40.1
Patrick Steinhardt· May 3, 2023, 11:34 UTC · re: Patrick Steinhardt · lore

[PATCH v3 5/8] fetch: introduce `display_format` enum

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a `enum display_format` that is more readily extensible.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)
Show changes to builtin/fetch.c +64 −37
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 6aecf549e8..9e7e45344d 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -784,7 +790,6 @@ static int refcol_width(const struct ref *ref, int compact_format)
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display format %d", display_state->format);
 	}
 }
 
@@ -904,30 +920,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
-- 
2.40.1
Patrick Steinhardt· May 3, 2023, 11:34 UTC · re: Patrick Steinhardt · lore

[PATCH v3 6/8] fetch: move display format parsing into main function

Parsing the display format happens inside of `display_state_init()`. As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a output format though that is intended to be parseable by machines, for example inside of a script. In that case it becomes a bit awkward of an interface if you have to call git-fetch(1) with the `fetch.output` config key set. We're thus going to introduce a new `--output-format` switch for git-fetch(1) so that the output format can be configured more directly.

This means we'll have to hook parsing of the display format into the command line options parser. Having the code to determine the actual output format scattered across two different sites is hard to reason about though.

Refactor the code such that callers are expected to pass the display format that is to be used into `display_state_init()`. This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
Show changes to builtin/fetch.c +24 −17
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 9e7e45344d..e15d43dc1e 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -788,14 +788,13 @@ static int refcol_width(const struct ref *ref, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -812,15 +811,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT: {
@@ -1614,7 +1604,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1700,7 +1691,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2078,7 +2069,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2145,7 +2137,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2157,6 +2149,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	int i;
 	const char *bundle_uri;
+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
@@ -2183,6 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
+		const char *format = "full";
+
+		git_config_get_string_tmp("fetch.output", &format);
+		if (!strcasecmp(format, "full"))
+			display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(format, "compact"))
+			display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", format);
+	}
+
 	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
 		recurse_submodules = recurse_submodules_cli;
 
@@ -2311,7 +2317,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   display_format);
 	} else {
 		int max_children = max_jobs;
 
-- 
2.40.1
Patrick Steinhardt· May 3, 2023, 11:34 UTC · re: Patrick Steinhardt · lore

[PATCH v3 7/8] fetch: move option related variables into main function

The options of git-fetch(1) which we pass to `parse_options()` are declared globally in `builtin/fetch.c`. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into `cmd_fetch()`. Move variables that were previously forced to be declared globally and which are only used by `cmd_fetch()` into function-local scope.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 197 ++++++++++++++++++++++++------------------------
 1 file changed, 100 insertions(+), 97 deletions(-)
Show changes to builtin/fetch.c +100 −97
diff --git a/builtin/fetch.c b/builtin/fetch.c
index e15d43dc1e..820ec7285c 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -75,13 +75,12 @@ static int fetch_prune_tags_config = -1; /* unspecified */
 static int prune_tags = -1; /* unspecified */
 #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
 
-static int all, append, dry_run, force, keep, multiple, update_head_ok;
+static int append, dry_run, force, keep, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
@@ -160,92 +153,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2147,13 +2054,109 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
-	int i;
 	const char *bundle_uri;
+	const char *submodule_prefix = "";
 	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
+	int all = 0, multiple = 0;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
-- 
2.40.1
Patrick Steinhardt· May 3, 2023, 11:34 UTC · re: Patrick Steinhardt · lore

[PATCH v3 8/8] fetch: introduce machine-parseable "porcelain" output format

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

``` <flag> <old-object-id> <new-object-id> <local-reference>\n ```

We assume two conditions which are generally true:
    - The old and new object IDs have fixed known widths and cannot
      contain spaces.
    - References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote.

In a similar spirit, a second restriction is that this cannot be used with `--recurse-submodules`. This is because any reference updates would be ambiguous without also printing the repository in which the update happens.

Considering that both multi-remote and submodule fetches are rather niche and likely not going to be useful for the majority of usecases these omissions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/fetch-options.txt |   6 ++
 Documentation/git-fetch.txt     |   9 +++
 builtin/fetch.c                 |  91 +++++++++++++++++++-----
 t/t5574-fetch-output.sh         | 120 ++++++++++++++++++++++++++++++++
 4 files changed, 208 insertions(+), 18 deletions(-)
Show changes to 4 files +208 −18

Documentation/fetch-options.txt, Documentation/git-fetch.txt, builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 622bd84768..02a5ae21ef 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -78,6 +78,12 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--porcelain::
+	Print the output to standard output in an easy-to-parse format for
+	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
++
+This is incompatible with `--recurse-submodules=[yes|on-demand]`.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt
index fba66f1460..f123139c58 100644
--- a/Documentation/git-fetch.txt
+++ b/Documentation/git-fetch.txt
@@ -204,6 +204,15 @@ representing the status of a single ref. Each line is of the form:
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+When using `--porcelain`, the output format is intended to be
+machine-parseable. In contrast to the human-readable output formats it
+thus prints to standard output instead of standard error. Each line is
+of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 820ec7285c..187c4d373c 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -52,6 +52,7 @@ enum display_format {
 	DISPLAY_FORMAT_UNKNOWN = 0,
 	DISPLAY_FORMAT_FULL,
 	DISPLAY_FORMAT_COMPACT,
+	DISPLAY_FORMAT_PORCELAIN,
 };
 
 struct display_state {
@@ -745,6 +746,9 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 		break;
 	}
+	case DISPLAY_FORMAT_PORCELAIN:
+		/* We don't need to precompute anything here. */
+		break;
 	default:
 		BUG("unexpected display format %d", display_state->format);
 	}
@@ -815,8 +819,12 @@ static void print_compact(struct display_state *display_state,
 static void display_ref_update(struct display_state *display_state, char code,
 			       const char *summary, const char *error,
 			       const char *remote, const char *local,
+			       const struct object_id *old_oid,
+			       const struct object_id *new_oid,
 			       int summary_width)
 {
+	FILE *f = stderr;
+
 	if (verbosity < 0)
 		return;
 
@@ -849,12 +857,17 @@ static void display_ref_update(struct display_state *display_state, char code,
 
 		break;
 	}
+	case DISPLAY_FORMAT_PORCELAIN:
+		strbuf_addf(&display_state->buf, "%c %s %s %s", code,
+			    oid_to_hex(old_oid), oid_to_hex(new_oid), local);
+		f = stdout;
+		break;
 	default:
 		BUG("unexpected display format %d", display_state->format);
 	};
 	strbuf_addch(&display_state->buf, '\n');
 
-	fputs(display_state->buf.buf, stderr);
+	fputs(display_state->buf.buf, f);
 }
 
 static int update_local_ref(struct ref *ref,
@@ -872,7 +885,8 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 		return 0;
 	}
 
@@ -885,7 +899,8 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 
@@ -896,12 +911,14 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return 1;
 		}
 	}
@@ -934,7 +951,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return r;
 	}
 
@@ -956,7 +974,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -968,12 +987,14 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 }
@@ -1214,7 +1235,9 @@ static int store_updated_refs(struct display_state *display_state,
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
 						   rm->name,
-						   "FETCH_HEAD", summary_width);
+						   "FETCH_HEAD",
+						   &rm->new_oid, &rm->old_oid,
+						   summary_width);
 			}
 		}
 	}
@@ -1354,6 +1377,7 @@ static int prune_refs(struct display_state *display_state,
 		for (ref = stale_refs; ref; ref = ref->next) {
 			display_ref_update(display_state, '-', _("[deleted]"), NULL,
 					   _("(none)"), ref->name,
+					   &ref->new_oid, &ref->old_oid,
 					   summary_width);
 			warn_dangling_symref(stderr, dangling_msg, ref->name);
 		}
@@ -1786,7 +1810,8 @@ static int add_remote_or_group(const char *name, struct string_list *list)
 	return 1;
 }
 
-static void add_options_to_argv(struct strvec *argv)
+static void add_options_to_argv(struct strvec *argv,
+				enum display_format format)
 {
 	if (dry_run)
 		strvec_push(argv, "--dry-run");
@@ -1822,6 +1847,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--ipv6");
 	if (!write_fetch_head)
 		strvec_push(argv, "--no-write-fetch-head");
+	if (format == DISPLAY_FORMAT_PORCELAIN)
+		strvec_pushf(argv, "--porcelain");
 }
 
 /* Fetch multiple remotes in parallel */
@@ -1830,6 +1857,7 @@ struct parallel_fetch_state {
 	const char **argv;
 	struct string_list *remotes;
 	int next, result;
+	enum display_format format;
 };
 
 static int fetch_next_remote(struct child_process *cp,
@@ -1849,7 +1877,7 @@ static int fetch_next_remote(struct child_process *cp,
 	strvec_push(&cp->args, remote);
 	cp->git_cmd = 1;
 
-	if (verbosity >= 0)
+	if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)
 		printf(_("Fetching %s\n"), remote);
 
 	return 1;
@@ -1881,7 +1909,8 @@ static int fetch_finished(int result, struct strbuf *out,
 	return 0;
 }
 
-static int fetch_multiple(struct string_list *list, int max_children)
+static int fetch_multiple(struct string_list *list, int max_children,
+			  enum display_format format)
 {
 	int i, result = 0;
 	struct strvec argv = STRVEC_INIT;
@@ -1894,10 +1923,10 @@ static int fetch_multiple(struct string_list *list, int max_children)
 
 	strvec_pushl(&argv, "fetch", "--append", "--no-auto-gc",
 		     "--no-write-commit-graph", NULL);
-	add_options_to_argv(&argv);
+	add_options_to_argv(&argv, format);
 
 	if (max_children != 1 && list->nr != 1) {
-		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
+		struct parallel_fetch_state state = { argv.v, list, 0, 0, format };
 		const struct run_process_parallel_opts opts = {
 			.tr2_category = "fetch",
 			.tr2_label = "parallel/fetch",
@@ -1921,7 +1950,7 @@ static int fetch_multiple(struct string_list *list, int max_children)
 
 			strvec_pushv(&cmd.args, argv.v);
 			strvec_push(&cmd.args, name);
-			if (verbosity >= 0)
+			if (verbosity >= 0 && format != DISPLAY_FORMAT_PORCELAIN)
 				printf(_("Fetching %s\n"), name);
 			cmd.git_cmd = 1;
 			if (run_command(&cmd)) {
@@ -2052,6 +2081,13 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	return exit_code;
 }
 
+static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
+{
+	enum display_format *format = opt->value;
+	*format = DISPLAY_FORMAT_PORCELAIN;
+	return 0;
+}
+
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	const char *bundle_uri;
@@ -2104,6 +2140,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
+			       PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
@@ -2222,6 +2260,23 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		fetch_config_from_gitmodules(sfjc, rs);
 	}
 
+	if (display_format == DISPLAY_FORMAT_PORCELAIN) {
+		switch (recurse_submodules_cli) {
+		case RECURSE_SUBMODULES_OFF:
+		case RECURSE_SUBMODULES_DEFAULT:
+			/*
+			 * Reference updates in submodules would be ambiguous
+			 * in porcelain mode, so we reject this combination.
+			 */
+			recurse_submodules = RECURSE_SUBMODULES_OFF;
+			break;
+
+		default:
+			die(_("options '%s' and '%s' cannot be used together"),
+			    "--porcelain", "--recurse-submodules");
+		}
+	}
+
 	if (negotiate_only && !negotiation_tip.nr)
 		die(_("--negotiate-only needs one or more --negotiation-tip=*"));
 
@@ -2341,7 +2396,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			max_children = fetch_parallel_config;
 
 		/* TODO should this also die if we have a previous partial-clone? */
-		result = fetch_multiple(&list, max_children);
+		result = fetch_multiple(&list, max_children, display_format);
 	}
 
 
@@ -2363,7 +2418,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		if (max_children < 0)
 			max_children = fetch_parallel_config;
 
-		add_options_to_argv(&options);
+		add_options_to_argv(&options, display_format);
 		result = fetch_submodules(the_repository,
 					  &options,
 					  submodule_prefix,
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index 6e0f7e0046..18949e9323 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -56,6 +56,102 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch porcelain output' '
+	test_when_finished "rm -rf porcelain" &&
+
+	# Set up a bunch of references that we can use to demonstrate different
+	# kinds of flag symbols in the output format.
+	MAIN_OLD=$(git rev-parse HEAD) &&
+	git branch "fast-forward" &&
+	git branch "deleted-branch" &&
+	git checkout -b force-updated &&
+	test_commit --no-tag force-update-old &&
+	FORCE_UPDATED_OLD=$(git rev-parse HEAD) &&
+	git checkout main &&
+
+	# Clone and pre-seed the repositories. We fetch references into two
+	# namespaces so that we can test that rejected and force-updated
+	# references are reported properly.
+	refspecs="refs/heads/*:refs/unforced/* +refs/heads/*:refs/forced/*" &&
+	git clone . porcelain &&
+	git -C porcelain fetch origin $refspecs &&
+
+	# Now that we have set up the client repositories we can change our
+	# local references.
+	git branch new-branch &&
+	git branch -d deleted-branch &&
+	git checkout fast-forward &&
+	test_commit --no-tag fast-forward-new &&
+	FAST_FORWARD_NEW=$(git rev-parse HEAD) &&
+	git checkout force-updated &&
+	git reset --hard HEAD~ &&
+	test_commit --no-tag force-update-new &&
+	FORCE_UPDATED_NEW=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	- $MAIN_OLD $ZERO_OID refs/forced/deleted-branch
+	- $MAIN_OLD $ZERO_OID refs/unforced/deleted-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/unforced/fast-forward
+	! $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/unforced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/unforced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/forced/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/forced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/forced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/remotes/origin/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/remotes/origin/force-updated
+	* $ZERO_OID $MAIN_OLD refs/remotes/origin/new-branch
+	EOF
+
+	# Execute a dry-run fetch first. We do this to assert that the dry-run
+	# and non-dry-run fetches produces the same output. Execution of the
+	# fetch is expected to fail as we have a rejected reference update.
+	test_must_fail git -C porcelain fetch \
+		--porcelain --dry-run --prune origin $refspecs >actual &&
+	test_cmp expect actual &&
+
+	# And now we perform a non-dry-run fetch.
+	test_must_fail git -C porcelain fetch \
+		--porcelain --prune origin $refspecs >actual 2>stderr &&
+	test_cmp expect actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'fetch porcelain with multiple remotes' '
+	test_when_finished "rm -rf porcelain" &&
+
+	git clone . porcelain &&
+	git -C porcelain remote add second-remote "$PWD" &&
+	git -C porcelain fetch second-remote &&
+
+	test_commit --no-tag multi-commit &&
+	old_commit=$(git rev-parse HEAD~) &&
+	new_commit=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	  $old_commit $new_commit refs/remotes/origin/force-updated
+	  $old_commit $new_commit refs/remotes/second-remote/force-updated
+	EOF
+
+	git -C porcelain fetch --porcelain --all >actual 2>stderr &&
+	test_cmp expect actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'fetch porcelain refuses to work with submodules' '
+	test_when_finished "rm -rf porcelain" &&
+
+	cat >expect <<-EOF &&
+	fatal: options ${SQ}--porcelain${SQ} and ${SQ}--recurse-submodules${SQ} cannot be used together
+	EOF
+
+	git init porcelain &&
+	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=yes 2>stderr &&
+	test_cmp expect stderr &&
+
+	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=on-demand 2>stderr &&
+	test_cmp expect stderr
+'
+
 test_expect_success 'fetch output with HEAD' '
 	test_when_finished "rm -rf head" &&
 	git clone . head &&
@@ -85,6 +181,30 @@ test_expect_success 'fetch output with HEAD' '
 	test_cmp expect actual.err
 '
 
+test_expect_success 'fetch porcelain output with HEAD' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+	COMMIT_ID=$(git rev-parse HEAD) &&
+
+	git -C head fetch --porcelain --dry-run origin HEAD >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --porcelain origin HEAD >actual &&
+	test_cmp expect actual &&
+
+	git -C head fetch --porcelain --dry-run origin HEAD:foo >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID refs/heads/foo
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --porcelain origin HEAD:foo >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
-- 
2.40.1
Glen Choo· May 8, 2023, 23:42 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v3 8/8] fetch: introduce machine-parseable "porcelain" output format

This version looks great! I only have minor comments.
Patrick Steinhardt <ps@pks.im> writes:
Show 17 quoted lines
> A notable ommission here is that the output format does not include the
> remote from which a reference was fetched, which might be important
> information especially in the context of multi-remote fetches. But as
> such a format would require us to print the remote for every single
> reference update due to parallelizable fetches it feels wasteful for the
> most likely usecase, which is when fetching from a single remote.
>
> In a similar spirit, a second restriction is that this cannot be used
> with `--recurse-submodules`. This is because any reference updates would
> be ambiguous without also printing the repository in which the update
> happens.
>
> Considering that both multi-remote and submodule fetches are rather
> niche and likely not going to be useful for the majority of usecases
> these omissions feel acceptable. If usecases for either of these come up
> in the future though it is easy enough to add a new "porcelain-v2"
> format that adds this information.

As a point of clarification, I think these options aren't niche in themselves, but they are more user-facing, so using them _in conjunction with_ --porcelain is probably pretty niche, so I think this is okay for now.

Show 34 quoted lines
> --- a/Documentation/fetch-options.txt
> +++ b/Documentation/fetch-options.txt
> @@ -78,6 +78,12 @@ linkgit:git-config[1].
>  --dry-run::
>  	Show what would be done, without making any changes.
>  
> +--porcelain::
> +	Print the output to standard output in an easy-to-parse format for
> +	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
> ++
> +This is incompatible with `--recurse-submodules=[yes|on-demand]`.
> +
>  ifndef::git-pull[]
>  --[no-]write-fetch-head::
>  	Write the list of remote refs fetched in the `FETCH_HEAD`
> diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt
> index fba66f1460..f123139c58 100644
> --- a/Documentation/git-fetch.txt
> +++ b/Documentation/git-fetch.txt
> @@ -204,6 +204,15 @@ representing the status of a single ref. Each line is of the form:
>   <flag> <summary> <from> -> <to> [<reason>]
>  -------------------------------
>  
> +When using `--porcelain`, the output format is intended to be
> +machine-parseable. In contrast to the human-readable output formats it
> +thus prints to standard output instead of standard error. Each line is
> +of the form:
> +
> +-------------------------------
> +<flag> <old-object-id> <new-object-id> <local-reference>
> +-------------------------------
> +
>  The status of up-to-date refs is shown only if the --verbose option is
>  used.
I hadn't commented on the docs before, but they are very clear. Thanks!
Show 17 quoted lines
> @@ -1830,6 +1857,7 @@ struct parallel_fetch_state {
>  	const char **argv;
>  	struct string_list *remotes;
>  	int next, result;
> +	enum display_format format;
>  };
>  
>  static int fetch_next_remote(struct child_process *cp,
> @@ -1849,7 +1877,7 @@ static int fetch_next_remote(struct child_process *cp,
>  	strvec_push(&cp->args, remote);
>  	cp->git_cmd = 1;
>  
> -	if (verbosity >= 0)
> +	if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)
>  		printf(_("Fetching %s\n"), remote);
>  
>  	return 1;

Here and elsewhere, I wonder if it's clearer to name the variable "porcelain" and separate it from "enum display_format". Then we can check "porcelain" directly instead of using "format == DISPLAY_FORMAT_PORCELAIN" as a proxy...

Show 19 quoted lines
> +static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
> +{
> +	enum display_format *format = opt->value;
> +	*format = DISPLAY_FORMAT_PORCELAIN;
> +	return 0;
> +}
> +
>  int cmd_fetch(int argc, const char **argv, const char *prefix)
>  {
>  	const char *bundle_uri;
> @@ -2104,6 +2140,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
>  			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
>  		OPT_BOOL(0, "dry-run", &dry_run,
>  			 N_("dry run")),
> +		OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
> +			       PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
>  		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
>  			 N_("write fetched references to the FETCH_HEAD file")),
>  		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),

e.g. since we are reusing the "display_format" variable, we need to make sure we parse "--porcelain" after we read "fetch.output". I double-checked to make sure we were doing the right thing, though it would be nice to not have to worry about those sorts of things. This shouldn't hold up the series though.

Show 15 quoted lines
> +test_expect_success 'fetch porcelain with multiple remotes' '
> +	test_when_finished "rm -rf porcelain" &&
> +
> +	git clone . porcelain &&
> +	git -C porcelain remote add second-remote "$PWD" &&
> +	git -C porcelain fetch second-remote &&
> +
> +	test_commit --no-tag multi-commit &&
> +	old_commit=$(git rev-parse HEAD~) &&
> +	new_commit=$(git rev-parse HEAD) &&
> +
> +	cat >expect <<-EOF &&
> +	  $old_commit $new_commit refs/remotes/origin/force-updated
> +	  $old_commit $new_commit refs/remotes/second-remote/force-updated
> +	EOF

The only thing in this test that relies on the previous test is that HEAD is pointing to "force-updated", and it's hard to tell where HEAD is since the previous test is so long. Could we create a new branch instead?

Patrick Steinhardt· May 9, 2023, 12:41 UTC · re: Glen Choo · lore

Re: [PATCH v3 8/8] fetch: introduce machine-parseable "porcelain" output format

On Mon, May 08, 2023 at 04:42:46PM -0700, Glen Choo wrote:
Show 26 quoted lines
> This version looks great! I only have minor comments.
> 
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > A notable ommission here is that the output format does not include the
> > remote from which a reference was fetched, which might be important
> > information especially in the context of multi-remote fetches. But as
> > such a format would require us to print the remote for every single
> > reference update due to parallelizable fetches it feels wasteful for the
> > most likely usecase, which is when fetching from a single remote.
> >
> > In a similar spirit, a second restriction is that this cannot be used
> > with `--recurse-submodules`. This is because any reference updates would
> > be ambiguous without also printing the repository in which the update
> > happens.
> >
> > Considering that both multi-remote and submodule fetches are rather
> > niche and likely not going to be useful for the majority of usecases
> > these omissions feel acceptable. If usecases for either of these come up
> > in the future though it is easy enough to add a new "porcelain-v2"
> > format that adds this information.
> 
> As a point of clarification, I think these options aren't niche in
> themselves, but they are more user-facing, so using them _in conjunction
> with_ --porcelain is probably pretty niche, so I think this is okay for
> now.
Yeah, that's what I indeed intended to say. Will clarify.
[snip]
Show 22 quoted lines
> > @@ -1830,6 +1857,7 @@ struct parallel_fetch_state {
> >  	const char **argv;
> >  	struct string_list *remotes;
> >  	int next, result;
> > +	enum display_format format;
> >  };
> >  
> >  static int fetch_next_remote(struct child_process *cp,
> > @@ -1849,7 +1877,7 @@ static int fetch_next_remote(struct child_process *cp,
> >  	strvec_push(&cp->args, remote);
> >  	cp->git_cmd = 1;
> >  
> > -	if (verbosity >= 0)
> > +	if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)
> >  		printf(_("Fetching %s\n"), remote);
> >  
> >  	return 1;
> 
> Here and elsewhere, I wonder if it's clearer to name the variable
> "porcelain" and separate it from "enum display_format". Then we can
> check "porcelain" directly instead of using "format ==
> DISPLAY_FORMAT_PORCELAIN" as a proxy...

For now I'd like to keep this as-is: it's easier to keep track of this when there is only a single variable that keeps track of the output format. But if we were to add additional porcelain formats in the future I agree that it would be nice to refactor the code as you propose.

I'd rather keep the simpler version for now though where we only have a single state to worry about.

Show 25 quoted lines
> > +static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
> > +{
> > +	enum display_format *format = opt->value;
> > +	*format = DISPLAY_FORMAT_PORCELAIN;
> > +	return 0;
> > +}
> > +
> >  int cmd_fetch(int argc, const char **argv, const char *prefix)
> >  {
> >  	const char *bundle_uri;
> > @@ -2104,6 +2140,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
> >  			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
> >  		OPT_BOOL(0, "dry-run", &dry_run,
> >  			 N_("dry run")),
> > +		OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
> > +			       PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
> >  		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
> >  			 N_("write fetched references to the FETCH_HEAD file")),
> >  		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
> 
> e.g. since we are reusing the "display_format" variable, we need to make
> sure we parse "--porcelain" after we read "fetch.output". I
> double-checked to make sure we were doing the right thing, though it
> would be nice to not have to worry about those sorts of things. This
> shouldn't hold up the series though.

In fact it is the other way round: we parse `--porcelain` first and then only read "fetch.output" in the case where the `display_format` variable is still set to `DISPLAY_FORMAT_UNKNOWN`.

So in the end there is no fragile order dependence here -- it would work just the same regardless of whether we first parse command line options or the configuration.

Show 20 quoted lines
> > +test_expect_success 'fetch porcelain with multiple remotes' '
> > +	test_when_finished "rm -rf porcelain" &&
> > +
> > +	git clone . porcelain &&
> > +	git -C porcelain remote add second-remote "$PWD" &&
> > +	git -C porcelain fetch second-remote &&
> > +
> > +	test_commit --no-tag multi-commit &&
> > +	old_commit=$(git rev-parse HEAD~) &&
> > +	new_commit=$(git rev-parse HEAD) &&
> > +
> > +	cat >expect <<-EOF &&
> > +	  $old_commit $new_commit refs/remotes/origin/force-updated
> > +	  $old_commit $new_commit refs/remotes/second-remote/force-updated
> > +	EOF
> 
> The only thing in this test that relies on the previous test is that
> HEAD is pointing to "force-updated", and it's hard to tell where HEAD is
> since the previous test is so long. Could we create a new branch
> instead?
Makes sense, done.
Patrick
Glen Choo· May 9, 2023, 00:03 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v3 8/8] fetch: introduce machine-parseable "porcelain" output format

Patrick Steinhardt <ps@pks.im> writes:
Show 6 quoted lines
> +--porcelain::
> +	Print the output to standard output in an easy-to-parse format for
> +	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
> ++
> +This is incompatible with `--recurse-submodules=[yes|on-demand]`.
> +

We should probably specify that this will override any setting of "fetch.output" (and perhaps add a test for it). Most readers will probably guess that this is the case, but the extra clarity wouldn't hurt.

Junio C Hamano· May 3, 2023, 16:48 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v3 0/8] fetch: introduce machine-parseable output

Patrick Steinhardt <ps@pks.im> writes:
> this is the third version of my patch series to introduce a
> machine-parseable output for git-fetch(1).

Which base did you pick to build these 8 patches? It seems that 6/8 is not happy when the series is attempted to be applied on top of many recent tips of 'master' that I tried (including 667fcf4e15 that the previous iteration was queued on top of in my tree).

Thanks.
Junio C Hamano· May 3, 2023, 16:53 UTC · re: Junio C Hamano · lore

Re: [PATCH v3 0/8] fetch: introduce machine-parseable output

Junio C Hamano <gitster@pobox.com> writes:
Show 9 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
>
>> this is the third version of my patch series to introduce a
>> machine-parseable output for git-fetch(1).
>
> Which base did you pick to build these 8 patches?  It seems that 6/8
> is not happy when the series is attempted to be applied on top of
> many recent tips of 'master' that I tried (including 667fcf4e15 that
> the previous iteration was queued on top of in my tree).

OK, I found e9dffbc7 (Merge branch 'ps/fetch-ref-update-reporting', 2023-04-06). I can manage from here.

Thanks.
Patrick Steinhardt· May 4, 2023, 07:57 UTC · re: Junio C Hamano · lore

Re: [PATCH v3 0/8] fetch: introduce machine-parseable output

On Wed, May 03, 2023 at 09:53:04AM -0700, Junio C Hamano wrote:
Show 16 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
> 
> > Patrick Steinhardt <ps@pks.im> writes:
> >
> >> this is the third version of my patch series to introduce a
> >> machine-parseable output for git-fetch(1).
> >
> > Which base did you pick to build these 8 patches?  It seems that 6/8
> > is not happy when the series is attempted to be applied on top of
> > many recent tips of 'master' that I tried (including 667fcf4e15 that
> > the previous iteration was queued on top of in my tree).
> 
> OK, I found e9dffbc7 (Merge branch 'ps/fetch-ref-update-reporting',
> 2023-04-06).  I can manage from here.
> 
> Thanks.

Yeah, that's where I based it off from. Sorry for not explicitly mentioning this fact.

Patrick
Glen Choo· May 9, 2023, 00:06 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v3 0/8] fetch: introduce machine-parseable output

Patrick Steinhardt <ps@pks.im> writes:
> Hi,
>
> this is the third version of my patch series to introduce a
> machine-parseable output for git-fetch(1).

This version looks great! I only had minor comments this time, mainly on docs and tests.

Thanks for your patience and receptiveness, and for the fruitful discussions :)

Patrick Steinhardt· May 9, 2023, 12:42 UTC · re: Glen Choo · lore

Re: [PATCH v3 0/8] fetch: introduce machine-parseable output

On Mon, May 08, 2023 at 05:06:28PM -0700, Glen Choo wrote:
Show 12 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Hi,
> >
> > this is the third version of my patch series to introduce a
> > machine-parseable output for git-fetch(1).
> 
> This version looks great! I only had minor comments this time, mainly on
> docs and tests.
> 
> Thanks for your patience and receptiveness, and for the fruitful
> discussions :)
Thank _you_ for the extensive feedback and constructive criticism.
Patrick
Patrick Steinhardt· May 9, 2023, 13:01 UTC · re: Patrick Steinhardt · lore

[PATCH v4 0/8] fetch: introduce machine-parseable output

Hi,

this is the fourth version of my patch series to introduce a machine-parseable output format for git-fetch(1). It applies on top of e9dffbc7f1 (Merge branch 'ps/fetch-ref-update-reporting', 2023-04-06).

There's only a small set of change compared to v3:
    - Patch 1/8: The test added by this change has been significantly
      simplified.
    - Patch 8/8: Reworded the commit message to clarify that parallel
      fetches and `--recurse-submodules` aren't niche on their own, but
      are likely niche combined with `--porcelain` given that they are
      user-facing while `--porcelain` is script-facing.
    - Patch 8/8: Clarified that `--porcelain` takes precedence over the
      `fetch.output` config option in the user-facing documentation.
      I've also added a test to verify that this is the case.
    - Patch 8/8: Amended one test to not depend on the state that its
      preceding test cretaes.
Thanks to Glen for the feedback.
Patrick
Patrick Steinhardt (8):
  fetch: fix `--no-recurse-submodules` with multi-remote fetches
  fetch: split out tests for output format
  fetch: add a test to exercise invalid output formats
  fetch: fix missing from-reference when fetching HEAD:foo
  fetch: introduce `display_format` enum
  fetch: move display format parsing into main function
  fetch: move option related variables into main function
  fetch: introduce machine-parseable "porcelain" output format
 Documentation/fetch-options.txt |   7 +
 Documentation/git-fetch.txt     |   9 +
 builtin/fetch.c                 | 437 +++++++++++++++++++-------------
 t/t5510-fetch.sh                |  53 ----
 t/t5526-fetch-submodules.sh     |  19 ++
 t/t5574-fetch-output.sh         | 251 ++++++++++++++++++
 6 files changed, 552 insertions(+), 224 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Range-diff against v3:
1:  4b2b0cfe15 ! 1:  d82b42ed34 fetch: fix `--no-recurse-submodules` with multi-remote fetches
    @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetch --all with --recurse-sub
      '
      
     +test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
    -+	test_when_finished "git -C downstream remote remove second" &&
    ++	test_when_finished "rm -rf src_clone" &&
     +
    -+	# We need to add a second remote, otherwise --all falls back to the
    -+	# normal fetch-one case.
    -+	git -C downstream remote add second .. &&
    -+	git -C downstream fetch --all &&
    ++	git clone --recurse-submodules src src_clone &&
    ++	(
    ++		cd src_clone &&
    ++		git remote add secondary ../src &&
    ++		git config submodule.recurse true &&
    ++		git config fetch.parallel 0 &&
    ++		git fetch --all --no-recurse-submodules 2>../actual
    ++	) &&
     +
    -+	add_submodule_commits &&
    -+	add_superproject_commits &&
    -+	old_commit=$(git rev-parse --short HEAD~) &&
    -+	new_commit=$(git rev-parse --short HEAD) &&
    -+
    -+	git -C downstream fetch --all --no-recurse-submodules >actual.out 2>actual.err &&
    -+
    -+	cat >expect.out <<-EOF &&
    -+	Fetching origin
    -+	Fetching second
    ++	cat >expect <<-EOF &&
    ++	From ../src
    ++	 * [new branch]      master     -> secondary/master
     +	EOF
    -+
    -+	cat >expect.err <<-EOF &&
    -+	From $(test-tool path-utils real_path .)/.
    -+	   $old_commit..$new_commit  super      -> origin/super
    -+	From ..
    -+	   $old_commit..$new_commit  super      -> second/super
    -+	EOF
    -+
    -+	test_cmp expect.out actual.out &&
    -+	test_cmp expect.err actual.err
    ++	test_cmp expect actual
     +'
     +
      test_done
2:  6ebc7450ba = 2:  33112dc51a fetch: split out tests for output format
3:  78479922ac = 3:  006ea93afb fetch: add a test to exercise invalid output formats
4:  46e1266ab0 = 4:  e599ea6d33 fetch: fix missing from-reference when fetching HEAD:foo
5:  acc0f7f520 = 5:  80ac00b0c4 fetch: introduce `display_format` enum
6:  cd23440128 = 6:  826b8b7bc0 fetch: move display format parsing into main function
7:  edbc31013f = 7:  20f2e061d6 fetch: move option related variables into main function
8:  e132d9494e ! 8:  24ae381950 fetch: introduce machine-parseable "porcelain" output format
    @@ Commit message
         be ambiguous without also printing the repository in which the update
         happens.
     
    -    Considering that both multi-remote and submodule fetches are rather
    -    niche and likely not going to be useful for the majority of usecases
    -    these omissions feel acceptable. If usecases for either of these come up
    -    in the future though it is easy enough to add a new "porcelain-v2"
    -    format that adds this information.
    +    Considering that both multi-remote and submodule fetches are user-facing
    +    features, using them in conjunction with `--porcelain` that is intended
    +    for scripting purposes is likely not going to be useful in the majority
    +    of cases. With that in mind these restrictions feel acceptable. If
    +    usecases for either of these come up in the future though it is easy
    +    enough to add a new "porcelain-v2" format that adds this information.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    @@ Documentation/fetch-options.txt: linkgit:git-config[1].
     +	Print the output to standard output in an easy-to-parse format for
     +	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
     ++
    -+This is incompatible with `--recurse-submodules=[yes|on-demand]`.
    ++This is incompatible with `--recurse-submodules=[yes|on-demand]` and takes
    ++precedence over the `fetch.output` config option.
     +
      ifndef::git-pull[]
      --[no-]write-fetch-head::
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +test_expect_success 'fetch porcelain with multiple remotes' '
     +	test_when_finished "rm -rf porcelain" &&
     +
    ++	git switch --create multiple-remotes &&
     +	git clone . porcelain &&
     +	git -C porcelain remote add second-remote "$PWD" &&
     +	git -C porcelain fetch second-remote &&
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +	new_commit=$(git rev-parse HEAD) &&
     +
     +	cat >expect <<-EOF &&
    -+	  $old_commit $new_commit refs/remotes/origin/force-updated
    -+	  $old_commit $new_commit refs/remotes/second-remote/force-updated
    ++	  $old_commit $new_commit refs/remotes/origin/multiple-remotes
    ++	  $old_commit $new_commit refs/remotes/second-remote/multiple-remotes
     +	EOF
     +
     +	git -C porcelain fetch --porcelain --all >actual 2>stderr &&
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=on-demand 2>stderr &&
     +	test_cmp expect stderr
     +'
    ++
    ++test_expect_success 'fetch porcelain overrides fetch.output config' '
    ++	test_when_finished "rm -rf porcelain" &&
    ++
    ++	git switch --create config-override &&
    ++	git clone . porcelain &&
    ++	test_commit new-commit &&
    ++	old_commit=$(git rev-parse HEAD~) &&
    ++	new_commit=$(git rev-parse HEAD) &&
    ++
    ++	cat >expect <<-EOF &&
    ++	  $old_commit $new_commit refs/remotes/origin/config-override
    ++	* $ZERO_OID $new_commit refs/tags/new-commit
    ++	EOF
    ++
    ++	git -C porcelain -c fetch.output=compact fetch --porcelain >stdout 2>stderr &&
    ++	test_must_be_empty stderr &&
    ++	test_cmp expect stdout
    ++'
     +
      test_expect_success 'fetch output with HEAD' '
      	test_when_finished "rm -rf head" &&
-- 
2.40.1
Patrick Steinhardt· May 9, 2023, 13:02 UTC · re: Patrick Steinhardt · lore

[PATCH v4 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

When running `git fetch --no-recurse-submodules`, the exectation is that we don't fetch any submodules. And while this works for fetches of a single remote, it doesn't when fetching multiple remotes at once. The result is that we do recurse into submodules even though the user has explicitly asked us not to.

This is because while we pass on `--recurse-submodules={yes,on-demand}` if specified by the user, we don't pass on `--no-recurse-submodules` to the subprocess spawned to perform the submodule fetch.

Fix this by also forwarding this flag as expected.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c             |  2 ++
 t/t5526-fetch-submodules.sh | 19 +++++++++++++++++++
 2 files changed, 21 insertions(+)
Show changes to 2 files +21 −0

builtin/fetch.c, t/t5526-fetch-submodules.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index c310d89878..08d7fc7233 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1876,6 +1876,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--keep");
 	if (recurse_submodules == RECURSE_SUBMODULES_ON)
 		strvec_push(argv, "--recurse-submodules");
+	else if (recurse_submodules == RECURSE_SUBMODULES_OFF)
+		strvec_push(argv, "--no-recurse-submodules");
 	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
 		strvec_push(argv, "--recurse-submodules=on-demand");
 	if (tags == TAGS_SET)
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index dcdbe26a08..ba69cd583f 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -1180,4 +1180,23 @@ test_expect_success 'fetch --all with --recurse-submodules with multiple' '
 	test_line_count = 2 fetch-subs
 '
 
+test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
+	test_when_finished "rm -rf src_clone" &&
+
+	git clone --recurse-submodules src src_clone &&
+	(
+		cd src_clone &&
+		git remote add secondary ../src &&
+		git config submodule.recurse true &&
+		git config fetch.parallel 0 &&
+		git fetch --all --no-recurse-submodules 2>../actual
+	) &&
+
+	cat >expect <<-EOF &&
+	From ../src
+	 * [new branch]      master     -> secondary/master
+	EOF
+	test_cmp expect actual
+'
+
 test_done
-- 
2.40.1
Junio C Hamano· May 9, 2023, 17:49 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v4 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

Patrick Steinhardt <ps@pks.im> writes:
Show 11 quoted lines
> When running `git fetch --no-recurse-submodules`, the exectation is that
> we don't fetch any submodules. And while this works for fetches of a
> single remote, it doesn't when fetching multiple remotes at once. The
> result is that we do recurse into submodules even though the user has
> explicitly asked us not to.
>
> This is because while we pass on `--recurse-submodules={yes,on-demand}`
> if specified by the user, we don't pass on `--no-recurse-submodules` to
> the subprocess spawned to perform the submodule fetch.
>
> Fix this by also forwarding this flag as expected.
Makes sense.
Show 8 quoted lines
> +test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
> +	test_when_finished "rm -rf src_clone" &&
> +
> +	git clone --recurse-submodules src src_clone &&
> +	(
> +		cd src_clone &&
> +		git remote add secondary ../src &&
> +		git config submodule.recurse true &&

The above two is essential to this test; we are interested in making sure that --no-recurse-submodules is propagated down even when the "--all" option is used, and we want another remote for that. We set the default to recurse, so that passing "--no-recurse-submodules" would defeat it, but just refraining to pass "--recurse-submodules" would cause us to recurse.

> +		git config fetch.parallel 0 &&

Is this necessary for the purpose of the test, though? It should not hurt, but we do not require the end-users to set it in real life for the parallel fetching to work, either, right?

Just being curious.
Show 9 quoted lines
> +		git fetch --all --no-recurse-submodules 2>../actual
> +	) &&
> +
> +	cat >expect <<-EOF &&
> +	From ../src
> +	 * [new branch]      master     -> secondary/master
> +	EOF
> +	test_cmp expect actual
> +'

In the context of a series that attempts to introduce a new stable output format for machine consumption, which implies the traditional output can change to match human users' preference, this test feels a bit brittle, but let's wait until the end of the series to judge that.

Looking good.  Thanks.
>  test_done
Glen Choo· May 9, 2023, 18:27 UTC · re: Junio C Hamano · lore

Re: [PATCH v4 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

Junio C Hamano <gitster@pobox.com> writes:
Show 5 quoted lines
>> +		git config fetch.parallel 0 &&
>
> Is this necessary for the purpose of the test, though?  It should
> not hurt, but we do not require the end-users to set it in real life
> for the parallel fetching to work, either, right?

IIUC it would make the test output deterministic if we fetched from both remotes. That doesn't happen here though, so I guess it's not doing anything right now.

Show 15 quoted lines
>> +		git fetch --all --no-recurse-submodules 2>../actual
>> +	) &&
>> +
>> +	cat >expect <<-EOF &&
>> +	From ../src
>> +	 * [new branch]      master     -> secondary/master
>> +	EOF
>> +	test_cmp expect actual
>> +'
>
> In the context of a series that attempts to introduce a new stable
> output format for machine consumption, which implies the traditional
> output can change to match human users' preference, this test feels
> a bit brittle, but let's wait until the end of the series to judge
> that.

I also find it a bit brittle to assert on the whole output when this test is about checking that we do not fetch the superproject.

Is there a reason you didn't go with the "grep for submodule lines" approach in the previous tests? If it's about catching regressions, IMO your PATCH 2/8 does a good enough job of doing that.

Wondering out loud, I wonder if it makes sense for us to make a bigger distinction between "tests whose purpose is to guard against unexpected changes in output" (i.e. snapshot tests) vs "tests that happen to use output as a way to assert behavior" (i.e. 'regular' behavioral tests). Many GUI app codebases have such a distinction and have different best practices around them.

Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Glen Choo · lore

Re: [PATCH v4 1/8] fetch: fix `--no-recurse-submodules` with multi-remote fetches

On Tue, May 09, 2023 at 11:27:35AM -0700, Glen Choo wrote:
Show 11 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
> 
> >> +		git config fetch.parallel 0 &&
> >
> > Is this necessary for the purpose of the test, though?  It should
> > not hurt, but we do not require the end-users to set it in real life
> > for the parallel fetching to work, either, right?
> 
> IIUC it would make the test output deterministic if we fetched from both
> remotes. That doesn't happen here though, so I guess it's not doing
> anything right now.
Right, will drop.
Show 29 quoted lines
> >> +		git fetch --all --no-recurse-submodules 2>../actual
> >> +	) &&
> >> +
> >> +	cat >expect <<-EOF &&
> >> +	From ../src
> >> +	 * [new branch]      master     -> secondary/master
> >> +	EOF
> >> +	test_cmp expect actual
> >> +'
> >
> > In the context of a series that attempts to introduce a new stable
> > output format for machine consumption, which implies the traditional
> > output can change to match human users' preference, this test feels
> > a bit brittle, but let's wait until the end of the series to judge
> > that.
> 
> I also find it a bit brittle to assert on the whole output when this
> test is about checking that we do not fetch the superproject.
> 
> Is there a reason you didn't go with the "grep for submodule lines"
> approach in the previous tests? If it's about catching regressions, IMO
> your PATCH 2/8 does a good enough job of doing that.
> 
> Wondering out loud, I wonder if it makes sense for us to make a bigger
> distinction between "tests whose purpose is to guard against unexpected
> changes in output" (i.e. snapshot tests) vs "tests that happen to use
> output as a way to assert behavior" (i.e. 'regular' behavioral tests).
> Many GUI app codebases have such a distinction and have different best
> practices around them.
Fair enough, will switch to `! grep "Fetching submodule"`.
Patrick
Patrick Steinhardt· May 9, 2023, 13:02 UTC · re: Patrick Steinhardt · lore

[PATCH v4 2/8] fetch: split out tests for output format

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Show changes to 2 files +63 −53

t/t5510-fetch.sh, t/t5574-fetch-output.sh

diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index dc44da9c79..4f289063ce 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
new file mode 100755
index 0000000000..f91b654d38
--- /dev/null
+++ b/t/t5574-fetch-output.sh
@@ -0,0 +1,63 @@
+#!/bin/sh
+
+test_description='git fetch output format'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+
+test_expect_success 'fetch aligned output' '
+	git clone . full-output &&
+	test_commit looooooooooooong-tag &&
+	(
+		cd full-output &&
+		git -c fetch.output=full fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main                 -> origin/main
+	looooooooooooong-tag -> looooooooooooong-tag
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'fetch compact output' '
+	git clone . compact &&
+	test_commit extraaa &&
+	(
+		cd compact &&
+		git -c fetch.output=compact fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main       -> origin/*
+	extraaa    -> *
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success '--no-show-forced-updates' '
+	mkdir forced-updates &&
+	(
+		cd forced-updates &&
+		git init &&
+		test_commit 1 &&
+		test_commit 2
+	) &&
+	git clone forced-updates forced-update-clone &&
+	git clone forced-updates no-forced-update-clone &&
+	git -C forced-updates reset --hard HEAD~1 &&
+	(
+		cd forced-update-clone &&
+		git fetch --show-forced-updates origin 2>output &&
+		test_i18ngrep "(forced update)" output
+	) &&
+	(
+		cd no-forced-update-clone &&
+		git fetch --no-show-forced-updates origin 2>output &&
+		test_i18ngrep ! "(forced update)" output
+	)
+'
+
+test_done
-- 
2.40.1
Patrick Steinhardt· May 9, 2023, 13:02 UTC · re: Patrick Steinhardt · lore

[PATCH v4 3/8] fetch: add a test to exercise invalid output formats

Add a testcase that exercises the logic when an invalid output format is passed via the `fetch.output` configuration.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5574-fetch-output.sh | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
Show changes to t/t5574-fetch-output.sh +19 −0
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index f91b654d38..a09750d225 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -7,6 +7,25 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
-- 
2.40.1
Junio C Hamano· May 9, 2023, 17:58 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v4 3/8] fetch: add a test to exercise invalid output formats

Patrick Steinhardt <ps@pks.im> writes:
Show 7 quoted lines
> Add a testcase that exercises the logic when an invalid output format is
> passed via the `fetch.output` configuration.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  t/t5574-fetch-output.sh | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)

It makes perfect sense to make sure that invalid input gets rejected and the command exits with non-zero status, and it is probably a good thing that the end-user sees a message that explains why the particular input is rejected (even though it adds one more thing that needs to be updated when the message gets reworded).

But do we need to insist on no output to the standard output stream when the command errors out?

Other than that, looking good.
THanks.
Show 30 quoted lines
> diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
> index f91b654d38..a09750d225 100755
> --- a/t/t5574-fetch-output.sh
> +++ b/t/t5574-fetch-output.sh
> @@ -7,6 +7,25 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
>  
>  . ./test-lib.sh
>  
> +test_expect_success 'fetch with invalid output format configuration' '
> +	test_when_finished "rm -rf clone" &&
> +	git clone . clone &&
> +
> +	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
> +	cat >expect <<-EOF &&
> +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
> +	EOF
> +	test_must_be_empty actual.out &&
> +	test_cmp expect actual.err &&
> +
> +	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
> +	cat >expect <<-EOF &&
> +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
> +	EOF
> +	test_must_be_empty actual.out &&
> +	test_cmp expect actual.err
> +'
> +
>  test_expect_success 'fetch aligned output' '
>  	git clone . full-output &&
>  	test_commit looooooooooooong-tag &&
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Junio C Hamano · lore

Re: [PATCH v4 3/8] fetch: add a test to exercise invalid output formats

On Tue, May 09, 2023 at 10:58:33AM -0700, Junio C Hamano wrote:
Show 20 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Add a testcase that exercises the logic when an invalid output format is
> > passed via the `fetch.output` configuration.
> >
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >  t/t5574-fetch-output.sh | 19 +++++++++++++++++++
> >  1 file changed, 19 insertions(+)
> 
> It makes perfect sense to make sure that invalid input gets rejected
> and the command exits with non-zero status, and it is probably a
> good thing that the end-user sees a message that explains why the
> particular input is rejected (even though it adds one more thing
> that needs to be updated when the message gets reworded).
> 
> But do we need to insist on no output to the standard output stream
> when the command errors out?
> 
> Other than that, looking good.
Not really, no. Will change.
Patrick
Patrick Steinhardt· May 9, 2023, 13:02 UTC · re: Patrick Steinhardt · lore

[PATCH v4 4/8] fetch: fix missing from-reference when fetching HEAD:foo

`store_updated_refs()` parses the remote reference for two purposes:
    - It gets used as a note when writing FETCH_HEAD.
    - It is passed through to `display_ref_update()` to display
      updated references in the following format:
      ```
       * branch               master          -> master
      ```

In most cases, the parsed remote reference is the prettified reference name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case. But it is not intended when displaying the updated references and would cause us to miss the left-hand side of the displayed reference update:

```
$ git fetch origin HEAD:foo
From https://github.com/git/git
 * [new ref]                          -> foo
```

The HEAD string is clearly missing from the left-hand side of the arrow, which is further stressed by the point that the following commands show the left-hand side as expected:

```
$ git fetch origin HEAD
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD
$ git fetch origin master
From https://github.com/git/git
 * branch                  master     -> FETCH_HEAD
 * branch                  master     -> origin/master
```

The logic of how we compute the remote reference name that we ultimately pass to `display_ref_update()` is not easy to follow. There are three different cases here:

    - When the remote reference name is "HEAD" we set the remote
      reference name to the empty string. This is the case that causes
      the bug to occur, where we would indeed want to print "HEAD"
      instead of the empty string. This is what `prettify_refname()`
      would return.
    - When the remote reference name has a well-known prefix then we
      strip this prefix. This matches what `prettify_refname()` does.
    - Otherwise, we keep the fully qualified reference name. This also
      matches what `prettify_refname()` does.

As the return value of `prettify_refname()` would do the correct thing for us in all three cases, we can fix the bug by passing through the full remote reference name to `display_ref_update()`, which learns to call `prettify_refname()`. At the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" and "what" variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c         | 37 +++++++++++++++++++------------------
 t/t5574-fetch-output.sh | 29 +++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+), 18 deletions(-)
Show changes to 2 files +48 −18

builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 08d7fc7233..6aecf549e8 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index a09750d225..6e0f7e0046 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -56,6 +56,35 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD:foo >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
-- 
2.40.1
Junio C Hamano· May 9, 2023, 19:28 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v4 4/8] fetch: fix missing from-reference when fetching HEAD:foo

Patrick Steinhardt <ps@pks.im> writes:
Show 16 quoted lines
> But it is not intended when displaying the updated references and would
> cause us to miss the left-hand side of the displayed reference update:
>
> ```
> $ git fetch origin HEAD:foo
> From https://github.com/git/git
>  * [new ref]                          -> foo
> ```
> The HEAD string is clearly missing from the left-hand side of the arrow,
> which is further stressed by the point that the following commands show
> the left-hand side as expected:
>
> ```
> $ git fetch origin HEAD
> From https://github.com/git/git
>  * branch                  HEAD       -> FETCH_HEAD

I do not mind being explicit and showing HEAD in this case for the sake of consistency.

But speaking for the past developers, it was deliberate to omit what is common from the output to make it more terse, IIRC, and I think it is unfair to call it a "BUG".

Back when we wrote git-fetch-script, the output was a lot more verbose, and through efforts like 165f3902 (git-fetch: more terse fetch output, 2007-11-03) and numerous others over time, we got to the current output.

> Note that this patch also changes formatting of the block that computes
> the "kind" and "what" variables. This is done on purpose so that it is
> part of the diff, hopefully making the change easier to comprehend.

Just to help readers, "kind" is the category like branch, tag, etc. and "what" is the concrete name like 'master' and 'foo'.

Show 36 quoted lines
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 08d7fc7233..6aecf549e8 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
>  	}
>  
>  	width = (summary_width + strlen(summary) - gettext_width(summary));
> +	remote = prettify_refname(remote);
> +	local = prettify_refname(local);
>  
>  	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
>  	if (!display_state->compact_format)
> -		print_remote_to_local(display_state, remote, prettify_refname(local));
> +		print_remote_to_local(display_state, remote, local);
>  	else
> -		print_compact(display_state, remote, prettify_refname(local));
> +		print_compact(display_state, remote, local);
>  	if (error)
>  		strbuf_addf(&display_state->buf, "  (%s)", error);
>  	strbuf_addch(&display_state->buf, '\n');
> @@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
>  static int update_local_ref(struct ref *ref,
>  			    struct ref_transaction *transaction,
>  			    struct display_state *display_state,
> -			    const char *remote, const struct ref *remote_ref,
> +			    const struct ref *remote_ref,
>  			    int summary_width)
>  {
>  	struct commit *current = NULL, *updated;
> @@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
>  	if (oideq(&ref->old_oid, &ref->new_oid)) {
>  		if (verbosity > 0)
>  			display_ref_update(display_state, '=', _("[up to date]"), NULL,
> -					   remote, ref->name, summary_width);
> +					   remote_ref->name, ref->name, summary_width);

Makes sense. The variable "remote" (now removed) holds what to write to FETCH_HEAD to be used to formulate a merge message by the caller, but this function is purely to report the ref updates and has no need to have access to that information.

Show 19 quoted lines
> @@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
>  			if (!strcmp(rm->name, "HEAD")) {
>  				kind = "";
>  				what = "";
> -			}
> -			else if (skip_prefix(rm->name, "refs/heads/", &what))
> +			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
>  				kind = "branch";
> -			else if (skip_prefix(rm->name, "refs/tags/", &what))
> +			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
>  				kind = "tag";
> -			else if (skip_prefix(rm->name, "refs/remotes/", &what))
> +			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
>  				kind = "remote-tracking branch";
> -			else {
> +			} else {
>  				kind = "";
>  				what = rm->name;
>  			}

This is a bit noisier than necessary. It took me a while until I realized that this hunk is a no-op.

Show 8 quoted lines
> @@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
>  					  display_state->url_len);
>  
>  			if (ref) {
> -				rc |= update_local_ref(ref, transaction, display_state, what,
> +				rc |= update_local_ref(ref, transaction, display_state,
>  						       rm, summary_width);
>  				free(ref);
Good.
Show 7 quoted lines
> @@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
>  				 */
>  				display_ref_update(display_state, '*',
>  						   *kind ? kind : "branch", NULL,
> -						   *what ? what : "HEAD",
> +						   rm->name,
>  						   "FETCH_HEAD", summary_width);

Good, too. The original cleared "what" and then to compensate for that had a logic to turn it back to "HEAD", but that is all gone by passing rm->name down.

I think we could pass "rm" and leave it to display_ref_update() what string to use, if we wanted to further refine the output later.

Then somebody in the future may even want to see "HEAD" to be shown as an empty string and that can all be done in display_ref_update(). It would fix the inconsistency that "git fetch origin HEAD" reports "HEAD -> FETCH_HEAD" by hiding "HEAD" just like the case where fetching "HEAD:foo" does, going in the other direction, I would think.

Thanks.
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Junio C Hamano · lore

Re: [PATCH v4 4/8] fetch: fix missing from-reference when fetching HEAD:foo

On Tue, May 09, 2023 at 12:28:16PM -0700, Junio C Hamano wrote:
Show 30 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > But it is not intended when displaying the updated references and would
> > cause us to miss the left-hand side of the displayed reference update:
> >
> > ```
> > $ git fetch origin HEAD:foo
> > From https://github.com/git/git
> >  * [new ref]                          -> foo
> > ```
> > The HEAD string is clearly missing from the left-hand side of the arrow,
> > which is further stressed by the point that the following commands show
> > the left-hand side as expected:
> >
> > ```
> > $ git fetch origin HEAD
> > From https://github.com/git/git
> >  * branch                  HEAD       -> FETCH_HEAD
> 
> I do not mind being explicit and showing HEAD in this case for the
> sake of consistency.
> 
> But speaking for the past developers, it was deliberate to omit what
> is common from the output to make it more terse, IIRC, and I think
> it is unfair to call it a "BUG".
> 
> Back when we wrote git-fetch-script, the output was a lot more
> verbose, and through efforts like 165f3902 (git-fetch: more terse
> fetch output, 2007-11-03) and numerous others over time, we got to
> the current output.

That's fair. It's still not quite clear whether this behaviour is in fact intentional though. Quoting 165f3902 (git-fetch: more terse fetch output, 2007-11-03), the weird corner case is not documented:

    This makes the fetch output much more terse and prettier on a 80 column
    display, based on a consensus reached on the mailing list.  Here's an
    example output:
    Receiving objects: 100% (5439/5439), 1.60 MiB | 636 KiB/s, done.
    Resolving deltas: 100% (4604/4604), done.
    From git://git.kernel.org/pub/scm/git/git
     ! [rejected]        html -> origin/html  (non fast forward)
       136e631..f45e867  maint -> origin/maint  (fast forward)
       9850e2e..44dd7e0  man -> origin/man  (fast forward)
       3e4bb08..e3d6d56  master -> origin/master  (fast forward)
       fa3665c..536f64a  next -> origin/next  (fast forward)
     + 4f6d9d6...768326f pu -> origin/pu  (forced update)
     * [new branch]      todo -> origin/todo

I've reformulated the commit message to talk about an inconsistency instead of a bug.

Patrick
Patrick Steinhardt· May 9, 2023, 13:02 UTC · re: Patrick Steinhardt · lore

[PATCH v4 5/8] fetch: introduce `display_format` enum

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a `enum display_format` that is more readily extensible.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 101 ++++++++++++++++++++++++++++++------------------
 1 file changed, 64 insertions(+), 37 deletions(-)
Show changes to builtin/fetch.c +64 −37
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 6aecf549e8..9e7e45344d 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -784,7 +790,6 @@ static int refcol_width(const struct ref *ref, int compact_format)
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		struct ref *rm;
 
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
+		display_state->refcol_width = 10;
+		for (rm = ref_map; rm; rm = rm->next) {
+			int width;
 
-		width = refcol_width(rm, display_state->compact_format);
+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
+			    !rm->peer_ref ||
+			    !strcmp(rm->name, "HEAD"))
+				continue;
 
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
+
+			/*
+			 * Not precise calculation for compact mode because '*' can
+			 * appear on the left hand side of '->' and shrink the column
+			 * back.
+			 */
+			if (display_state->refcol_width < width)
+				display_state->refcol_width = width;
+		}
+
+		break;
+	}
+	default:
+		BUG("unexpected display format %d", display_state->format);
 	}
 }
 
@@ -904,30 +920,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
-- 
2.40.1
Junio C Hamano· May 9, 2023, 20:19 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v4 5/8] fetch: introduce `display_format` enum

Patrick Steinhardt <ps@pks.im> writes:
Show 8 quoted lines
> We currently have two different display formats in git-fetch(1) with the
> "full" and "compact" formats. This is tracked with a boolean value that
> simply denotes whether the display format is supposed to be compacted
> or not. This works reasonably well while there are only two formats, but
> we're about to introduce another format that will make this a bit more
> awkward to use.
>
> Introduce a `enum display_format` that is more readily extensible.
Makes sense.
Show 12 quoted lines
> +enum display_format {
> +	DISPLAY_FORMAT_UNKNOWN = 0,
> +	DISPLAY_FORMAT_FULL,
> +	DISPLAY_FORMAT_COMPACT,
> +};
>
>  struct display_state {
>  	struct strbuf buf;
>  
>  	int refcol_width;
> -	int compact_format;
> +	enum display_format format;
OK.  Preparatory conversion without adding anything new.
Show 12 quoted lines
> @@ -809,31 +814,42 @@ static void display_state_init(struct display_state *display_state, struct ref *
>  
>  	git_config_get_string_tmp("fetch.output", &format);
>  	if (!strcasecmp(format, "full"))
> -		display_state->compact_format = 0;
> +		display_state->format = DISPLAY_FORMAT_FULL;
>  	else if (!strcasecmp(format, "compact"))
> -		display_state->compact_format = 1;
> +		display_state->format = DISPLAY_FORMAT_COMPACT;
>  	else
>  		die(_("invalid value for '%s': '%s'"),
>  		    "fetch.output", format);
Naturally.
Show 45 quoted lines
> -	display_state->refcol_width = 10;
> -	for (rm = ref_map; rm; rm = rm->next) {
> -		int width;
> +	switch (display_state->format) {
> +	case DISPLAY_FORMAT_FULL:
> +	case DISPLAY_FORMAT_COMPACT: {
> +		struct ref *rm;
>  
> -		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
> -		    !rm->peer_ref ||
> -		    !strcmp(rm->name, "HEAD"))
> -			continue;
> +		display_state->refcol_width = 10;
> +		for (rm = ref_map; rm; rm = rm->next) {
> +			int width;
>  
> -		width = refcol_width(rm, display_state->compact_format);
> +			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
> +			    !rm->peer_ref ||
> +			    !strcmp(rm->name, "HEAD"))
> +				continue;
>  
> -		/*
> -		 * Not precise calculation for compact mode because '*' can
> -		 * appear on the left hand side of '->' and shrink the column
> -		 * back.
> -		 */
> -		if (display_state->refcol_width < width)
> -			display_state->refcol_width = width;
> +			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
> +
> +			/*
> +			 * Not precise calculation for compact mode because '*' can
> +			 * appear on the left hand side of '->' and shrink the column
> +			 * back.
> +			 */
> +			if (display_state->refcol_width < width)
> +				display_state->refcol_width = width;
> +		}
> +
> +		break;
> +	}
> +	default:
> +		BUG("unexpected display format %d", display_state->format);
>  	}

Due to reindentation, the patch is noisier than what it does (which should be "nothing, other than allowing another value in the .format member").

It makes me wonder if it would make it easier to read to move the bulk of this code to a helper function. If we are to give a name to what is being done in the above hunk, what would it be? It computes display->refcol_width in which all records would fit, but presumably if we are to add more things to be shown per ref and align them in a simlar way, we would compute widths for these other things there as well. Perhaps compute_display_alignment() or somesuch?

Patrick Steinhardt· May 10, 2023, 12:35 UTC · re: Junio C Hamano · lore

Re: [PATCH v4 5/8] fetch: introduce `display_format` enum

On Tue, May 09, 2023 at 01:19:29PM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
[snip]
Show 57 quoted lines
> > -	display_state->refcol_width = 10;
> > -	for (rm = ref_map; rm; rm = rm->next) {
> > -		int width;
> > +	switch (display_state->format) {
> > +	case DISPLAY_FORMAT_FULL:
> > +	case DISPLAY_FORMAT_COMPACT: {
> > +		struct ref *rm;
> >  
> > -		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
> > -		    !rm->peer_ref ||
> > -		    !strcmp(rm->name, "HEAD"))
> > -			continue;
> > +		display_state->refcol_width = 10;
> > +		for (rm = ref_map; rm; rm = rm->next) {
> > +			int width;
> >  
> > -		width = refcol_width(rm, display_state->compact_format);
> > +			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
> > +			    !rm->peer_ref ||
> > +			    !strcmp(rm->name, "HEAD"))
> > +				continue;
> >  
> > -		/*
> > -		 * Not precise calculation for compact mode because '*' can
> > -		 * appear on the left hand side of '->' and shrink the column
> > -		 * back.
> > -		 */
> > -		if (display_state->refcol_width < width)
> > -			display_state->refcol_width = width;
> > +			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
> > +
> > +			/*
> > +			 * Not precise calculation for compact mode because '*' can
> > +			 * appear on the left hand side of '->' and shrink the column
> > +			 * back.
> > +			 */
> > +			if (display_state->refcol_width < width)
> > +				display_state->refcol_width = width;
> > +		}
> > +
> > +		break;
> > +	}
> > +	default:
> > +		BUG("unexpected display format %d", display_state->format);
> >  	}
> 
> Due to reindentation, the patch is noisier than what it does (which
> should be "nothing, other than allowing another value in the .format
> member").
> 
> It makes me wonder if it would make it easier to read to move the
> bulk of this code to a helper function.  If we are to give a name to
> what is being done in the above hunk, what would it be?  It computes
> display->refcol_width in which all records would fit, but presumably
> if we are to add more things to be shown per ref and align them in a
> simlar way, we would compute widths for these other things there as
> well.  Perhaps compute_display_alignment() or somesuch?

The code already has such a function and calls it `refcol_width()`, but it only computes the width for a single reference. The most natural thing to do here would thus be to merge the loop over the references into that function. This would also allow us to skip some weirdness, like the fact that we skip some references inside `refcol_width()` while we skip other references in the `ref_map` loop.

This refactoring is also quite noisy, but it makes the code simpler overall and will make the patch introducing the enum less so.

Patrick
Patrick Steinhardt· May 9, 2023, 13:02 UTC · re: Patrick Steinhardt · lore

[PATCH v4 6/8] fetch: move display format parsing into main function

Parsing the display format happens inside of `display_state_init()`. As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a output format though that is intended to be parseable by machines, for example inside of a script. In that case it becomes a bit awkward of an interface if you have to call git-fetch(1) with the `fetch.output` config key set. We're thus going to introduce a new `--output-format` switch for git-fetch(1) so that the output format can be configured more directly.

This means we'll have to hook parsing of the display format into the command line options parser. Having the code to determine the actual output format scattered across two different sites is hard to reason about though.

Refactor the code such that callers are expected to pass the display format that is to be used into `display_state_init()`. This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
Show changes to builtin/fetch.c +24 −17
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 9e7e45344d..e15d43dc1e 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -788,14 +788,13 @@ static int refcol_width(const struct ref *ref, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -812,15 +811,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT: {
@@ -1614,7 +1604,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1700,7 +1691,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2078,7 +2069,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2145,7 +2137,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2157,6 +2149,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	int i;
 	const char *bundle_uri;
+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
 	int result = 0;
@@ -2183,6 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix,
 			     builtin_fetch_options, builtin_fetch_usage, 0);
 
+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
+		const char *format = "full";
+
+		git_config_get_string_tmp("fetch.output", &format);
+		if (!strcasecmp(format, "full"))
+			display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(format, "compact"))
+			display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", format);
+	}
+
 	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
 		recurse_submodules = recurse_submodules_cli;
 
@@ -2311,7 +2317,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   display_format);
 	} else {
 		int max_children = max_jobs;
 
-- 
2.40.1
Junio C Hamano· May 9, 2023, 20:35 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v4 6/8] fetch: move display format parsing into main function

Patrick Steinhardt <ps@pks.im> writes:
Show 6 quoted lines
> We're about to introduce a output format though that is intended to be
> parseable by machines, for example inside of a script. In that case it
> becomes a bit awkward of an interface if you have to call git-fetch(1)
> with the `fetch.output` config key set. We're thus going to introduce a
> new `--output-format` switch for git-fetch(1) so that the output format
> can be configured more directly.

Good. I was wondering about that code in the context of the previous patch, especially the error message had a hard-coded assumption that it comes from a configuration variable.

And the changes to display_state_init() to lift the responsibility of finding and validating .format out of the function, and the changes to intermediate functions to pass the .format through the callchain, are all expected and there was nothing questionable.

Show 24 quoted lines
> @@ -2157,6 +2149,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
>  {
>  	int i;
>  	const char *bundle_uri;
> +	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
>  	struct string_list list = STRING_LIST_INIT_DUP;
>  	struct remote *remote = NULL;
>  	int result = 0;
> @@ -2183,6 +2176,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
>  	argc = parse_options(argc, argv, prefix,
>  			     builtin_fetch_options, builtin_fetch_usage, 0);
>  
> +	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
> +		const char *format = "full";
> +
> +		git_config_get_string_tmp("fetch.output", &format);
> +		if (!strcasecmp(format, "full"))
> +			display_format = DISPLAY_FORMAT_FULL;
> +		else if (!strcasecmp(format, "compact"))
> +			display_format = DISPLAY_FORMAT_COMPACT;
> +		else
> +			die(_("invalid value for '%s': '%s'"),
> +			    "fetch.output", format);
> +	}

OK, but isn't the usual way to do this to have configuration parser before parse_options() and then let parse_options() override whatever display_format set by it?

That way, we do not have to worry about DISPLAY_FORMAT_UNKNOWN at all. Just initialize the variable to whatever default format at the beginning of this function, read "fetch.output" to override it if the configuration exists, and then let parse_options() to handle "--output-format" or "--porcelain" or whatever to further override it.

Thanks.
Glen Choo· May 9, 2023, 22:30 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v4 6/8] fetch: move display format parsing into main function

Patrick Steinhardt <ps@pks.im> writes:
> with the `fetch.output` config key set. We're thus going to introduce a
> new `--output-format` switch for git-fetch(1) so that the output format
> can be configured more directly.

This is stale as of v3, since isn't named --output-format any more. Let me see if there are other instances of this.

(I should have caught this earlier; I only saw it because Junio quoted it in a reply of his.)

Patrick Steinhardt· May 10, 2023, 12:35 UTC · re: Glen Choo · lore

Re: [PATCH v4 6/8] fetch: move display format parsing into main function

On Tue, May 09, 2023 at 03:30:33PM -0700, Glen Choo wrote:
Show 11 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > with the `fetch.output` config key set. We're thus going to introduce a
> > new `--output-format` switch for git-fetch(1) so that the output format
> > can be configured more directly.
> 
> This is stale as of v3, since isn't named --output-format any more. Let
> me see if there are other instances of this.
> 
> (I should have caught this earlier; I only saw it because Junio quoted
> it in a reply of his.)
Good catch, fixed now.
Patrick
Patrick Steinhardt· May 9, 2023, 13:02 UTC · re: Patrick Steinhardt · lore

[PATCH v4 7/8] fetch: move option related variables into main function

The options of git-fetch(1) which we pass to `parse_options()` are declared globally in `builtin/fetch.c`. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into `cmd_fetch()`. Move variables that were previously forced to be declared globally and which are only used by `cmd_fetch()` into function-local scope.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 197 ++++++++++++++++++++++++------------------------
 1 file changed, 100 insertions(+), 97 deletions(-)
Show changes to builtin/fetch.c +100 −97
diff --git a/builtin/fetch.c b/builtin/fetch.c
index e15d43dc1e..820ec7285c 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -75,13 +75,12 @@ static int fetch_prune_tags_config = -1; /* unspecified */
 static int prune_tags = -1; /* unspecified */
 #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
 
-static int all, append, dry_run, force, keep, multiple, update_head_ok;
+static int append, dry_run, force, keep, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
@@ -160,92 +153,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2147,13 +2054,109 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
-	int i;
 	const char *bundle_uri;
+	const char *submodule_prefix = "";
 	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
+	int all = 0, multiple = 0;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
-- 
2.40.1
Patrick Steinhardt· May 9, 2023, 13:02 UTC · re: Patrick Steinhardt · lore

[PATCH v4 8/8] fetch: introduce machine-parseable "porcelain" output format

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

``` <flag> <old-object-id> <new-object-id> <local-reference>\n ```

We assume two conditions which are generally true:
    - The old and new object IDs have fixed known widths and cannot
      contain spaces.
    - References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote.

In a similar spirit, a second restriction is that this cannot be used with `--recurse-submodules`. This is because any reference updates would be ambiguous without also printing the repository in which the update happens.

Considering that both multi-remote and submodule fetches are user-facing features, using them in conjunction with `--porcelain` that is intended for scripting purposes is likely not going to be useful in the majority of cases. With that in mind these restrictions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/fetch-options.txt |   7 ++
 Documentation/git-fetch.txt     |   9 ++
 builtin/fetch.c                 |  91 +++++++++++++++++----
 t/t5574-fetch-output.sh         | 140 ++++++++++++++++++++++++++++++++
 4 files changed, 229 insertions(+), 18 deletions(-)
Show changes to 4 files +229 −18

Documentation/fetch-options.txt, Documentation/git-fetch.txt, builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 622bd84768..41fc7ca3c6 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -78,6 +78,13 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--porcelain::
+	Print the output to standard output in an easy-to-parse format for
+	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
++
+This is incompatible with `--recurse-submodules=[yes|on-demand]` and takes
+precedence over the `fetch.output` config option.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt
index fba66f1460..f123139c58 100644
--- a/Documentation/git-fetch.txt
+++ b/Documentation/git-fetch.txt
@@ -204,6 +204,15 @@ representing the status of a single ref. Each line is of the form:
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+When using `--porcelain`, the output format is intended to be
+machine-parseable. In contrast to the human-readable output formats it
+thus prints to standard output instead of standard error. Each line is
+of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 820ec7285c..187c4d373c 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -52,6 +52,7 @@ enum display_format {
 	DISPLAY_FORMAT_UNKNOWN = 0,
 	DISPLAY_FORMAT_FULL,
 	DISPLAY_FORMAT_COMPACT,
+	DISPLAY_FORMAT_PORCELAIN,
 };
 
 struct display_state {
@@ -745,6 +746,9 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 		break;
 	}
+	case DISPLAY_FORMAT_PORCELAIN:
+		/* We don't need to precompute anything here. */
+		break;
 	default:
 		BUG("unexpected display format %d", display_state->format);
 	}
@@ -815,8 +819,12 @@ static void print_compact(struct display_state *display_state,
 static void display_ref_update(struct display_state *display_state, char code,
 			       const char *summary, const char *error,
 			       const char *remote, const char *local,
+			       const struct object_id *old_oid,
+			       const struct object_id *new_oid,
 			       int summary_width)
 {
+	FILE *f = stderr;
+
 	if (verbosity < 0)
 		return;
 
@@ -849,12 +857,17 @@ static void display_ref_update(struct display_state *display_state, char code,
 
 		break;
 	}
+	case DISPLAY_FORMAT_PORCELAIN:
+		strbuf_addf(&display_state->buf, "%c %s %s %s", code,
+			    oid_to_hex(old_oid), oid_to_hex(new_oid), local);
+		f = stdout;
+		break;
 	default:
 		BUG("unexpected display format %d", display_state->format);
 	};
 	strbuf_addch(&display_state->buf, '\n');
 
-	fputs(display_state->buf.buf, stderr);
+	fputs(display_state->buf.buf, f);
 }
 
 static int update_local_ref(struct ref *ref,
@@ -872,7 +885,8 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 		return 0;
 	}
 
@@ -885,7 +899,8 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 
@@ -896,12 +911,14 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return 1;
 		}
 	}
@@ -934,7 +951,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return r;
 	}
 
@@ -956,7 +974,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -968,12 +987,14 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 }
@@ -1214,7 +1235,9 @@ static int store_updated_refs(struct display_state *display_state,
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
 						   rm->name,
-						   "FETCH_HEAD", summary_width);
+						   "FETCH_HEAD",
+						   &rm->new_oid, &rm->old_oid,
+						   summary_width);
 			}
 		}
 	}
@@ -1354,6 +1377,7 @@ static int prune_refs(struct display_state *display_state,
 		for (ref = stale_refs; ref; ref = ref->next) {
 			display_ref_update(display_state, '-', _("[deleted]"), NULL,
 					   _("(none)"), ref->name,
+					   &ref->new_oid, &ref->old_oid,
 					   summary_width);
 			warn_dangling_symref(stderr, dangling_msg, ref->name);
 		}
@@ -1786,7 +1810,8 @@ static int add_remote_or_group(const char *name, struct string_list *list)
 	return 1;
 }
 
-static void add_options_to_argv(struct strvec *argv)
+static void add_options_to_argv(struct strvec *argv,
+				enum display_format format)
 {
 	if (dry_run)
 		strvec_push(argv, "--dry-run");
@@ -1822,6 +1847,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--ipv6");
 	if (!write_fetch_head)
 		strvec_push(argv, "--no-write-fetch-head");
+	if (format == DISPLAY_FORMAT_PORCELAIN)
+		strvec_pushf(argv, "--porcelain");
 }
 
 /* Fetch multiple remotes in parallel */
@@ -1830,6 +1857,7 @@ struct parallel_fetch_state {
 	const char **argv;
 	struct string_list *remotes;
 	int next, result;
+	enum display_format format;
 };
 
 static int fetch_next_remote(struct child_process *cp,
@@ -1849,7 +1877,7 @@ static int fetch_next_remote(struct child_process *cp,
 	strvec_push(&cp->args, remote);
 	cp->git_cmd = 1;
 
-	if (verbosity >= 0)
+	if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)
 		printf(_("Fetching %s\n"), remote);
 
 	return 1;
@@ -1881,7 +1909,8 @@ static int fetch_finished(int result, struct strbuf *out,
 	return 0;
 }
 
-static int fetch_multiple(struct string_list *list, int max_children)
+static int fetch_multiple(struct string_list *list, int max_children,
+			  enum display_format format)
 {
 	int i, result = 0;
 	struct strvec argv = STRVEC_INIT;
@@ -1894,10 +1923,10 @@ static int fetch_multiple(struct string_list *list, int max_children)
 
 	strvec_pushl(&argv, "fetch", "--append", "--no-auto-gc",
 		     "--no-write-commit-graph", NULL);
-	add_options_to_argv(&argv);
+	add_options_to_argv(&argv, format);
 
 	if (max_children != 1 && list->nr != 1) {
-		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
+		struct parallel_fetch_state state = { argv.v, list, 0, 0, format };
 		const struct run_process_parallel_opts opts = {
 			.tr2_category = "fetch",
 			.tr2_label = "parallel/fetch",
@@ -1921,7 +1950,7 @@ static int fetch_multiple(struct string_list *list, int max_children)
 
 			strvec_pushv(&cmd.args, argv.v);
 			strvec_push(&cmd.args, name);
-			if (verbosity >= 0)
+			if (verbosity >= 0 && format != DISPLAY_FORMAT_PORCELAIN)
 				printf(_("Fetching %s\n"), name);
 			cmd.git_cmd = 1;
 			if (run_command(&cmd)) {
@@ -2052,6 +2081,13 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	return exit_code;
 }
 
+static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
+{
+	enum display_format *format = opt->value;
+	*format = DISPLAY_FORMAT_PORCELAIN;
+	return 0;
+}
+
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
 	const char *bundle_uri;
@@ -2104,6 +2140,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
+			       PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
@@ -2222,6 +2260,23 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		fetch_config_from_gitmodules(sfjc, rs);
 	}
 
+	if (display_format == DISPLAY_FORMAT_PORCELAIN) {
+		switch (recurse_submodules_cli) {
+		case RECURSE_SUBMODULES_OFF:
+		case RECURSE_SUBMODULES_DEFAULT:
+			/*
+			 * Reference updates in submodules would be ambiguous
+			 * in porcelain mode, so we reject this combination.
+			 */
+			recurse_submodules = RECURSE_SUBMODULES_OFF;
+			break;
+
+		default:
+			die(_("options '%s' and '%s' cannot be used together"),
+			    "--porcelain", "--recurse-submodules");
+		}
+	}
+
 	if (negotiate_only && !negotiation_tip.nr)
 		die(_("--negotiate-only needs one or more --negotiation-tip=*"));
 
@@ -2341,7 +2396,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			max_children = fetch_parallel_config;
 
 		/* TODO should this also die if we have a previous partial-clone? */
-		result = fetch_multiple(&list, max_children);
+		result = fetch_multiple(&list, max_children, display_format);
 	}
 
 
@@ -2363,7 +2418,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		if (max_children < 0)
 			max_children = fetch_parallel_config;
 
-		add_options_to_argv(&options);
+		add_options_to_argv(&options, display_format);
 		result = fetch_submodules(the_repository,
 					  &options,
 					  submodule_prefix,
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index 6e0f7e0046..695624ac08 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -56,6 +56,122 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch porcelain output' '
+	test_when_finished "rm -rf porcelain" &&
+
+	# Set up a bunch of references that we can use to demonstrate different
+	# kinds of flag symbols in the output format.
+	MAIN_OLD=$(git rev-parse HEAD) &&
+	git branch "fast-forward" &&
+	git branch "deleted-branch" &&
+	git checkout -b force-updated &&
+	test_commit --no-tag force-update-old &&
+	FORCE_UPDATED_OLD=$(git rev-parse HEAD) &&
+	git checkout main &&
+
+	# Clone and pre-seed the repositories. We fetch references into two
+	# namespaces so that we can test that rejected and force-updated
+	# references are reported properly.
+	refspecs="refs/heads/*:refs/unforced/* +refs/heads/*:refs/forced/*" &&
+	git clone . porcelain &&
+	git -C porcelain fetch origin $refspecs &&
+
+	# Now that we have set up the client repositories we can change our
+	# local references.
+	git branch new-branch &&
+	git branch -d deleted-branch &&
+	git checkout fast-forward &&
+	test_commit --no-tag fast-forward-new &&
+	FAST_FORWARD_NEW=$(git rev-parse HEAD) &&
+	git checkout force-updated &&
+	git reset --hard HEAD~ &&
+	test_commit --no-tag force-update-new &&
+	FORCE_UPDATED_NEW=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	- $MAIN_OLD $ZERO_OID refs/forced/deleted-branch
+	- $MAIN_OLD $ZERO_OID refs/unforced/deleted-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/unforced/fast-forward
+	! $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/unforced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/unforced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/forced/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/forced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/forced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/remotes/origin/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/remotes/origin/force-updated
+	* $ZERO_OID $MAIN_OLD refs/remotes/origin/new-branch
+	EOF
+
+	# Execute a dry-run fetch first. We do this to assert that the dry-run
+	# and non-dry-run fetches produces the same output. Execution of the
+	# fetch is expected to fail as we have a rejected reference update.
+	test_must_fail git -C porcelain fetch \
+		--porcelain --dry-run --prune origin $refspecs >actual &&
+	test_cmp expect actual &&
+
+	# And now we perform a non-dry-run fetch.
+	test_must_fail git -C porcelain fetch \
+		--porcelain --prune origin $refspecs >actual 2>stderr &&
+	test_cmp expect actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'fetch porcelain with multiple remotes' '
+	test_when_finished "rm -rf porcelain" &&
+
+	git switch --create multiple-remotes &&
+	git clone . porcelain &&
+	git -C porcelain remote add second-remote "$PWD" &&
+	git -C porcelain fetch second-remote &&
+
+	test_commit --no-tag multi-commit &&
+	old_commit=$(git rev-parse HEAD~) &&
+	new_commit=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	  $old_commit $new_commit refs/remotes/origin/multiple-remotes
+	  $old_commit $new_commit refs/remotes/second-remote/multiple-remotes
+	EOF
+
+	git -C porcelain fetch --porcelain --all >actual 2>stderr &&
+	test_cmp expect actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'fetch porcelain refuses to work with submodules' '
+	test_when_finished "rm -rf porcelain" &&
+
+	cat >expect <<-EOF &&
+	fatal: options ${SQ}--porcelain${SQ} and ${SQ}--recurse-submodules${SQ} cannot be used together
+	EOF
+
+	git init porcelain &&
+	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=yes 2>stderr &&
+	test_cmp expect stderr &&
+
+	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=on-demand 2>stderr &&
+	test_cmp expect stderr
+'
+
+test_expect_success 'fetch porcelain overrides fetch.output config' '
+	test_when_finished "rm -rf porcelain" &&
+
+	git switch --create config-override &&
+	git clone . porcelain &&
+	test_commit new-commit &&
+	old_commit=$(git rev-parse HEAD~) &&
+	new_commit=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	  $old_commit $new_commit refs/remotes/origin/config-override
+	* $ZERO_OID $new_commit refs/tags/new-commit
+	EOF
+
+	git -C porcelain -c fetch.output=compact fetch --porcelain >stdout 2>stderr &&
+	test_must_be_empty stderr &&
+	test_cmp expect stdout
+'
+
 test_expect_success 'fetch output with HEAD' '
 	test_when_finished "rm -rf head" &&
 	git clone . head &&
@@ -85,6 +201,30 @@ test_expect_success 'fetch output with HEAD' '
 	test_cmp expect actual.err
 '
 
+test_expect_success 'fetch porcelain output with HEAD' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+	COMMIT_ID=$(git rev-parse HEAD) &&
+
+	git -C head fetch --porcelain --dry-run origin HEAD >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --porcelain origin HEAD >actual &&
+	test_cmp expect actual &&
+
+	git -C head fetch --porcelain --dry-run origin HEAD:foo >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID refs/heads/foo
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --porcelain origin HEAD:foo >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
-- 
2.40.1
Junio C Hamano· May 9, 2023, 20:43 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v4 8/8] fetch: introduce machine-parseable "porcelain" output format

Patrick Steinhardt <ps@pks.im> writes:
Show 6 quoted lines
> Considering that both multi-remote and submodule fetches are user-facing
> features, using them in conjunction with `--porcelain` that is intended
> for scripting purposes is likely not going to be useful in the majority
> of cases. With that in mind these restrictions feel acceptable. If
> usecases for either of these come up in the future though it is easy
> enough to add a new "porcelain-v2" format that adds this information.

Two steps ago, the proposed log message still mentioned "--output-format", which may want to be proofread again and revised.

Show 17 quoted lines
> @@ -1786,7 +1810,8 @@ static int add_remote_or_group(const char *name, struct string_list *list)
>  	return 1;
>  }
>  
> -static void add_options_to_argv(struct strvec *argv)
> +static void add_options_to_argv(struct strvec *argv,
> +				enum display_format format)
>  {
>  	if (dry_run)
>  		strvec_push(argv, "--dry-run");
> @@ -1822,6 +1847,8 @@ static void add_options_to_argv(struct strvec *argv)
>  		strvec_push(argv, "--ipv6");
>  	if (!write_fetch_head)
>  		strvec_push(argv, "--no-write-fetch-head");
> +	if (format == DISPLAY_FORMAT_PORCELAIN)
> +		strvec_pushf(argv, "--porcelain");
>  }
Hmph.  

[PATCH 9/8] may want to also introduce and pass down the "--output-format=full/compact" option, but that is clearly outside of the scope of this step.

Show 6 quoted lines
> +static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
> +{
> +	enum display_format *format = opt->value;
> +	*format = DISPLAY_FORMAT_PORCELAIN;
> +	return 0;
> +}
Lack of "if (unset) ..." worries me.  Shouldn't the code allow
	git fetch --porcelain --no-porcelain
to defeat an earlier one and revert back to the default?

Other than that the changes in this step were mostly expected and I didn't spot anything glaringly wrong, but I didn't spend as much time as I did for the other steps.

Thanks.
Patrick Steinhardt· May 10, 2023, 12:35 UTC · re: Junio C Hamano · lore

Re: [PATCH v4 8/8] fetch: introduce machine-parseable "porcelain" output format

On Tue, May 09, 2023 at 01:43:41PM -0700, Junio C Hamano wrote:
Show 11 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Considering that both multi-remote and submodule fetches are user-facing
> > features, using them in conjunction with `--porcelain` that is intended
> > for scripting purposes is likely not going to be useful in the majority
> > of cases. With that in mind these restrictions feel acceptable. If
> > usecases for either of these come up in the future though it is easy
> > enough to add a new "porcelain-v2" format that adds this information.
> 
> Two steps ago, the proposed log message still mentioned "--output-format",
> which may want to be proofread again and revised.

It's a relict from previous iterations. I've fixed the preceding patch to talk about `--porcelain` instead.

Show 23 quoted lines
> > @@ -1786,7 +1810,8 @@ static int add_remote_or_group(const char *name, struct string_list *list)
> >  	return 1;
> >  }
> >  
> > -static void add_options_to_argv(struct strvec *argv)
> > +static void add_options_to_argv(struct strvec *argv,
> > +				enum display_format format)
> >  {
> >  	if (dry_run)
> >  		strvec_push(argv, "--dry-run");
> > @@ -1822,6 +1847,8 @@ static void add_options_to_argv(struct strvec *argv)
> >  		strvec_push(argv, "--ipv6");
> >  	if (!write_fetch_head)
> >  		strvec_push(argv, "--no-write-fetch-head");
> > +	if (format == DISPLAY_FORMAT_PORCELAIN)
> > +		strvec_pushf(argv, "--porcelain");
> >  }
> 
> Hmph.  
> 
> [PATCH 9/8] may want to also introduce and pass down the
> "--output-format=full/compact" option, but that is clearly outside
> of the scope of this step.

We don't have `--output-format` now though, but only the `--porcelain` switch. The only way to configure "full"/"compact" output formats is via the "fetch.output" config variable right now.

Show 12 quoted lines
> > +static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
> > +{
> > +	enum display_format *format = opt->value;
> > +	*format = DISPLAY_FORMAT_PORCELAIN;
> > +	return 0;
> > +}
> 
> Lack of "if (unset) ..." worries me.  Shouldn't the code allow
> 
> 	git fetch --porcelain --no-porcelain
> 
> to defeat an earlier one and revert back to the default?

We pass `PARSE_OPT_NOARG|PARSE_OPT_NONEG`, so this isn't really much of a concern. But yeah, it wouldn't allow `--no-porcelain`, and supporting it is trivial enough to do. I'll change this to a `OPT_BOOL()`.

Patrick
Patrick Steinhardt· May 10, 2023, 12:33 UTC · re: Patrick Steinhardt · lore

[PATCH v5 0/9] fetch: introduce machine-parseable output

Hi,

this is the fourth version of my patch series to introduce a machine-parseable output format for git-fetch(1). It applies on top of e9dffbc7f1 (Merge branch 'ps/fetch-ref-update-reporting', 2023-04-06).

Changes compared to v4:
    - Patch 1/9: Simplified the test as proposed by Junio and Glen.
    - Patch 3/9: Added a test to verify that `git fetch -c fetch.output`
      without a value set fails as expected. Also dropped the tests that
      checked whether stdout was empty.
    - Patch 4/9: Reformulated the commit message to treat the missing
      left-hand side of displayed references as an inconsistency instead
      of a bug. I've also added a testcase to verify that direct OID
      fetches continue to work as expected.
    - Patch 5/9: New patch that makes calculation of the table width for
      displayed reference updates self-contained in `refcol_width()`.
      This is a preparatory refactoring that should make patch 6/9
      easier to review.
    - Patch 7/9: Refactored the code to parse the "fetch.output" config
      variable inside of `git_fetch_config()` before we parse command
      line options. Also fixed that the commit message was still
      referring to `--output-format=porcelain` instead of the new
      `--porcelain` switch.
    - Patch 9/9: The `--porcelain` option is now a simple `OPT_BOOL()`
      that can be negated. Added a test that `--no-porcelain` works as
      expected.
Thanks for your feedback, Junio and Glen!
Patrick
Patrick Steinhardt (9):
  fetch: fix `--no-recurse-submodules` with multi-remote fetches
  fetch: split out tests for output format
  fetch: add a test to exercise invalid output formats
  fetch: print left-hand side when fetching HEAD:foo
  fetch: refactor calculation of the display table width
  fetch: introduce `display_format` enum
  fetch: lift up parsing of "fetch.output" config variable
  fetch: move option related variables into main function
  fetch: introduce machine-parseable "porcelain" output format
 Documentation/fetch-options.txt |   7 +
 Documentation/git-fetch.txt     |   9 +
 builtin/fetch.c                 | 490 +++++++++++++++++++-------------
 t/t5510-fetch.sh                |  53 ----
 t/t5526-fetch-submodules.sh     |  13 +
 t/t5574-fetch-output.sh         | 293 +++++++++++++++++++
 6 files changed, 611 insertions(+), 254 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Range-diff against v4:
 1:  d82b42ed34 !  1:  02ee4fab7e fetch: fix `--no-recurse-submodules` with multi-remote fetches
    @@ t/t5526-fetch-submodules.sh: test_expect_success 'fetch --all with --recurse-sub
     +		cd src_clone &&
     +		git remote add secondary ../src &&
     +		git config submodule.recurse true &&
    -+		git config fetch.parallel 0 &&
    -+		git fetch --all --no-recurse-submodules 2>../actual
    ++		git fetch --all --no-recurse-submodules 2>../fetch-log
     +	) &&
    -+
    -+	cat >expect <<-EOF &&
    -+	From ../src
    -+	 * [new branch]      master     -> secondary/master
    -+	EOF
    -+	test_cmp expect actual
    ++	! grep "Fetching submodule" fetch-log
     +'
     +
      test_done
 2:  33112dc51a =  2:  e459d8a1ec fetch: split out tests for output format
 3:  006ea93afb !  3:  d503c425fe fetch: add a test to exercise invalid output formats
    @@ t/t5574-fetch-output.sh: export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
     +	test_when_finished "rm -rf clone" &&
     +	git clone . clone &&
     +
    -+	test_must_fail git -C clone -c fetch.output= fetch origin >actual.out 2>actual.err &&
    ++	test_must_fail git -C clone -c fetch.output fetch origin 2>actual.err &&
    ++	cat >expect <<-EOF &&
    ++	error: missing value for ${SQ}fetch.output${SQ}
    ++	fatal: unable to parse ${SQ}fetch.output${SQ} from command-line config
    ++	EOF
    ++	test_cmp expect actual.err &&
    ++
    ++	test_must_fail git -C clone -c fetch.output= fetch origin 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
     +	EOF
    -+	test_must_be_empty actual.out &&
     +	test_cmp expect actual.err &&
     +
    -+	test_must_fail git -C clone -c fetch.output=garbage fetch origin >actual.out 2>actual.err &&
    ++	test_must_fail git -C clone -c fetch.output=garbage fetch origin 2>actual.err &&
     +	cat >expect <<-EOF &&
     +	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
     +	EOF
    -+	test_must_be_empty actual.out &&
     +	test_cmp expect actual.err
     +'
     +
 4:  e599ea6d33 !  4:  2cc7318697 fetch: fix missing from-reference when fetching HEAD:foo
    @@ Metadata
     Author: Patrick Steinhardt <ps@pks.im>
     
      ## Commit message ##
    -    fetch: fix missing from-reference when fetching HEAD:foo
    +    fetch: print left-hand side when fetching HEAD:foo
     
         `store_updated_refs()` parses the remote reference for two purposes:
     
    @@ Commit message
         name and can thus be used for both cases. But if the remote reference is
         HEAD, the parsed remote reference becomes empty. This is intended when
         we write the FETCH_HEAD, where we skip writing the note in that case.
    -    But it is not intended when displaying the updated references and would
    -    cause us to miss the left-hand side of the displayed reference update:
    +    But when displaying the updated references this leads to inconsistent
    +    output where the left-hand side of reference updates is missing in some
    +    cases:
     
         ```
    -    $ git fetch origin HEAD:foo
    -    From https://github.com/git/git
    -     * [new ref]                          -> foo
    -    ```
    -
    -    The HEAD string is clearly missing from the left-hand side of the arrow,
    -    which is further stressed by the point that the following commands show
    -    the left-hand side as expected:
    -
    -    ```
    -    $ git fetch origin HEAD
    +    $ git fetch origin HEAD HEAD:explicit-head :implicit-head main
         From https://github.com/git/git
          * branch                  HEAD       -> FETCH_HEAD
    -
    -    $ git fetch origin master
    -    From https://github.com/git/git
    -     * branch                  master     -> FETCH_HEAD
    -     * branch                  master     -> origin/master
    +     * [new ref]                          -> explicit-head
    +     * [new ref]                          -> implicit-head
    +     * branch                  main       -> FETCH_HEAD
         ```
     
    +    This behaviour has existed ever since the table-based output has been
    +    introduced for git-fetch(1) via 165f390250 (git-fetch: more terse fetch
    +    output, 2007-11-03) and was never explicitly documented either in the
    +    commit message or in any of our tests. So while it may not be a bug per
    +    se, it feels like a weird inconsistency and not like it was a concious
    +    design decision.
    +
         The logic of how we compute the remote reference name that we ultimately
         pass to `display_ref_update()` is not easy to follow. There are three
         different cases here:
     
             - When the remote reference name is "HEAD" we set the remote
               reference name to the empty string. This is the case that causes
    -          the bug to occur, where we would indeed want to print "HEAD"
    -          instead of the empty string. This is what `prettify_refname()`
    -          would return.
    +          the left-hand side to go missing, where we would indeed want to
    +          print "HEAD" instead of the empty string. This is what
    +          `prettify_refname()` would return.
     
             - When the remote reference name has a well-known prefix then we
               strip this prefix. This matches what `prettify_refname()` does.
    @@ Commit message
               matches what `prettify_refname()` does.
     
         As the return value of `prettify_refname()` would do the correct thing
    -    for us in all three cases, we can fix the bug by passing through the
    -    full remote reference name to `display_ref_update()`, which learns to
    -    call `prettify_refname()`. At the same time, this also simplifies the
    -    code a bit.
    +    for us in all three cases, we can thus fix the inconsistency by passing
    +    through the full remote reference name to `display_ref_update()`, which
    +    learns to call `prettify_refname()`. At the same time, this also
    +    simplifies the code a bit.
     
         Note that this patch also changes formatting of the block that computes
    -    the "kind" and "what" variables. This is done on purpose so that it is
    -    part of the diff, hopefully making the change easier to comprehend.
    +    the "kind" (which is the category like "branch" or "tag") and "what"
    +    (which is the prettified reference name like "master" or "v1.0")
    +    variables. This is done on purpose so that it is part of the diff,
    +    hopefully making the change easier to comprehend.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +	test_must_be_empty actual.out &&
     +	test_cmp expect actual.err
     +'
    ++
    ++test_expect_success 'fetch output with object ID' '
    ++	test_when_finished "rm -rf object-id" &&
    ++	git clone . object-id &&
    ++	commit=$(git rev-parse HEAD) &&
    ++
    ++	git -C object-id fetch --dry-run origin $commit:object-id >actual.out 2>actual.err &&
    ++	cat >expect <<-EOF &&
    ++	From $(test-tool path-utils real_path .)/.
    ++	 * [new ref]         $commit -> object-id
    ++	EOF
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err &&
    ++
    ++	git -C object-id fetch origin $commit:object-id >actual.out 2>actual.err &&
    ++	test_must_be_empty actual.out &&
    ++	test_cmp expect actual.err
    ++'
     +
      test_expect_success '--no-show-forced-updates' '
      	mkdir forced-updates &&
 -:  ---------- >  5:  bb1a591c2f fetch: refactor calculation of the display table width
 5:  80ac00b0c4 !  6:  3cac552f5f fetch: introduce `display_format` enum
    @@ builtin/fetch.c: enum {
      
      	char *url;
      	int url_len, shown_url;
    -@@ builtin/fetch.c: static int refcol_width(const struct ref *ref, int compact_format)
    - static void display_state_init(struct display_state *display_state, struct ref *ref_map,
    - 			       const char *raw_url)
    - {
    --	struct ref *rm;
    - 	const char *format = "full";
    - 	int i;
    - 
     @@ builtin/fetch.c: static void display_state_init(struct display_state *display_state, struct ref *
      
      	git_config_get_string_tmp("fetch.output", &format);
    @@ builtin/fetch.c: static void display_state_init(struct display_state *display_st
      		die(_("invalid value for '%s': '%s'"),
      		    "fetch.output", format);
      
    --	display_state->refcol_width = 10;
    --	for (rm = ref_map; rm; rm = rm->next) {
    --		int width;
    +-	display_state->refcol_width = refcol_width(ref_map, display_state->compact_format);
     +	switch (display_state->format) {
     +	case DISPLAY_FORMAT_FULL:
    -+	case DISPLAY_FORMAT_COMPACT: {
    -+		struct ref *rm;
    - 
    --		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
    --		    !rm->peer_ref ||
    --		    !strcmp(rm->name, "HEAD"))
    --			continue;
    -+		display_state->refcol_width = 10;
    -+		for (rm = ref_map; rm; rm = rm->next) {
    -+			int width;
    - 
    --		width = refcol_width(rm, display_state->compact_format);
    -+			if (rm->status == REF_STATUS_REJECT_SHALLOW ||
    -+			    !rm->peer_ref ||
    -+			    !strcmp(rm->name, "HEAD"))
    -+				continue;
    - 
    --		/*
    --		 * Not precise calculation for compact mode because '*' can
    --		 * appear on the left hand side of '->' and shrink the column
    --		 * back.
    --		 */
    --		if (display_state->refcol_width < width)
    --			display_state->refcol_width = width;
    -+			width = refcol_width(rm, display_state->format == DISPLAY_FORMAT_COMPACT);
    -+
    -+			/*
    -+			 * Not precise calculation for compact mode because '*' can
    -+			 * appear on the left hand side of '->' and shrink the column
    -+			 * back.
    -+			 */
    -+			if (display_state->refcol_width < width)
    -+				display_state->refcol_width = width;
    -+		}
    -+
    ++	case DISPLAY_FORMAT_COMPACT:
    ++		display_state->refcol_width = refcol_width(ref_map,
    ++							   display_state->format == DISPLAY_FORMAT_COMPACT);
     +		break;
    -+	}
     +	default:
     +		BUG("unexpected display format %d", display_state->format);
    - 	}
    ++	}
      }
      
    + static void display_state_release(struct display_state *display_state)
     @@ builtin/fetch.c: static void display_ref_update(struct display_state *display_state, char code,
      			       const char *remote, const char *local,
      			       int summary_width)
 6:  826b8b7bc0 !  7:  0c3dbcd09f fetch: move display format parsing into main function
    @@ Metadata
     Author: Patrick Steinhardt <ps@pks.im>
     
      ## Commit message ##
    -    fetch: move display format parsing into main function
    +    fetch: lift up parsing of "fetch.output" config variable
     
         Parsing the display format happens inside of `display_state_init()`. As
         we only need to check for a simple config entry, this is a natural
         location to put this code as it means that display-state logic is neatly
         contained in a single location.
     
    -    We're about to introduce a output format though that is intended to be
    -    parseable by machines, for example inside of a script. In that case it
    -    becomes a bit awkward of an interface if you have to call git-fetch(1)
    -    with the `fetch.output` config key set. We're thus going to introduce a
    -    new `--output-format` switch for git-fetch(1) so that the output format
    -    can be configured more directly.
    -
    -    This means we'll have to hook parsing of the display format into the
    -    command line options parser. Having the code to determine the actual
    -    output format scattered across two different sites is hard to reason
    -    about though.
    +    We're about to introduce a new "porcelain" output format though that is
    +    intended to be parseable by machines, for example inside of a script.
    +    This format can be enabled by passing the `--porcelain` switch to
    +    git-fetch(1). As a consequence, we'll have to add a second callsite that
    +    influences the output format, which will become awkward to handle.
     
         Refactor the code such that callers are expected to pass the display
         format that is to be used into `display_state_init()`. This allows us to
    @@ Commit message
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## builtin/fetch.c ##
    -@@ builtin/fetch.c: static int refcol_width(const struct ref *ref, int compact_format)
    +@@ builtin/fetch.c: static int fetch_write_commit_graph = -1;
    + static int stdin_refspecs = 0;
    + static int negotiate_only;
    + 
    ++struct fetch_config {
    ++	enum display_format display_format;
    ++};
    ++
    + static int git_fetch_config(const char *k, const char *v, void *cb)
    + {
    ++	struct fetch_config *fetch_config = cb;
    ++
    + 	if (!strcmp(k, "fetch.prune")) {
    + 		fetch_prune_config = git_config_bool(k, v);
    + 		return 0;
    +@@ builtin/fetch.c: static int git_fetch_config(const char *k, const char *v, void *cb)
    + 		return 0;
    + 	}
    + 
    ++	if (!strcmp(k, "fetch.output")) {
    ++		if (!v)
    ++			return config_error_nonbool(k);
    ++		else if (!strcasecmp(v, "full"))
    ++			fetch_config->display_format = DISPLAY_FORMAT_FULL;
    ++		else if (!strcasecmp(v, "compact"))
    ++			fetch_config->display_format = DISPLAY_FORMAT_COMPACT;
    ++		else
    ++			die(_("invalid value for '%s': '%s'"),
    ++			    "fetch.output", v);
    ++	}
    ++
    + 	return git_default_config(k, v, cb);
    + }
    + 
    +@@ builtin/fetch.c: static int refcol_width(const struct ref *ref_map, int compact_format)
      }
      
      static void display_state_init(struct display_state *display_state, struct ref *ref_map,
    @@ builtin/fetch.c: static void display_state_init(struct display_state *display_st
     -
      	switch (display_state->format) {
      	case DISPLAY_FORMAT_FULL:
    - 	case DISPLAY_FORMAT_COMPACT: {
    + 	case DISPLAY_FORMAT_COMPACT:
     @@ builtin/fetch.c: static int backfill_tags(struct display_state *display_state,
      }
      
    @@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const cha
      	sigchain_pop(SIGPIPE);
      	refspec_clear(&rs);
      	transport_disconnect(gtransport);
    -@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    +@@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv,
    + 
    + int cmd_fetch(int argc, const char **argv, const char *prefix)
      {
    ++	struct fetch_config config = {
    ++		.display_format = DISPLAY_FORMAT_FULL,
    ++	};
      	int i;
      	const char *bundle_uri;
    -+	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
      	struct string_list list = STRING_LIST_INIT_DUP;
    - 	struct remote *remote = NULL;
    - 	int result = 0;
     @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    - 	argc = parse_options(argc, argv, prefix,
    - 			     builtin_fetch_options, builtin_fetch_usage, 0);
    - 
    -+	if (display_format == DISPLAY_FORMAT_UNKNOWN) {
    -+		const char *format = "full";
    -+
    -+		git_config_get_string_tmp("fetch.output", &format);
    -+		if (!strcasecmp(format, "full"))
    -+			display_format = DISPLAY_FORMAT_FULL;
    -+		else if (!strcasecmp(format, "compact"))
    -+			display_format = DISPLAY_FORMAT_COMPACT;
    -+		else
    -+			die(_("invalid value for '%s': '%s'"),
    -+			    "fetch.output", format);
    -+	}
    -+
    - 	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
    - 		recurse_submodules = recurse_submodules_cli;
    + 		free(anon);
    + 	}
      
    +-	git_config(git_fetch_config, NULL);
    ++	git_config(git_fetch_config, &config);
    + 	if (the_repository->gitdir) {
    + 		prepare_repo_settings(the_repository);
    + 		the_repository->settings.command_requires_full_index = 0;
     @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      	} else if (remote) {
      		if (filter_options.choice || has_promisor_remote())
      			fetch_one_setup_partial(remote);
     -		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
     +		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
    -+				   display_format);
    ++				   config.display_format);
      	} else {
      		int max_children = max_jobs;
      
 7:  20f2e061d6 !  8:  8e33a08c35 fetch: move option related variables into main function
    @@ builtin/fetch.c: static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
     -static int stdin_refspecs = 0;
     -static int negotiate_only;
      
    - static int git_fetch_config(const char *k, const char *v, void *cb)
    - {
    + struct fetch_config {
    + 	enum display_format display_format;
     @@ builtin/fetch.c: static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
      	return 0;
      }
    @@ builtin/fetch.c: static int parse_refmap_arg(const struct option *opt, const cha
      static void unlock_pack(unsigned int flags)
      {
      	if (gtransport)
    -@@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv,
    - 
    - int cmd_fetch(int argc, const char **argv, const char *prefix)
    - {
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 	struct fetch_config config = {
    + 		.display_format = DISPLAY_FORMAT_FULL,
    + 	};
     -	int i;
    - 	const char *bundle_uri;
     +	const char *submodule_prefix = "";
    - 	enum display_format display_format = DISPLAY_FORMAT_UNKNOWN;
    + 	const char *bundle_uri;
      	struct string_list list = STRING_LIST_INIT_DUP;
      	struct remote *remote = NULL;
     +	int all = 0, multiple = 0;
 8:  24ae381950 !  9:  d49152c220 fetch: introduce machine-parseable "porcelain" output format
    @@ builtin/fetch.c: enum display_format {
      
      struct display_state {
     @@ builtin/fetch.c: static void display_state_init(struct display_state *display_state, struct ref *
    - 
    + 		display_state->refcol_width = refcol_width(ref_map,
    + 							   display_state->format == DISPLAY_FORMAT_COMPACT);
      		break;
    - 	}
     +	case DISPLAY_FORMAT_PORCELAIN:
     +		/* We don't need to precompute anything here. */
     +		break;
    @@ builtin/fetch.c: static int fetch_multiple(struct string_list *list, int max_chi
      				printf(_("Fetching %s\n"), name);
      			cmd.git_cmd = 1;
      			if (run_command(&cmd)) {
    -@@ builtin/fetch.c: static int fetch_one(struct remote *remote, int argc, const char **argv,
    - 	return exit_code;
    - }
    +@@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 	int fetch_write_commit_graph = -1;
    + 	int stdin_refspecs = 0;
    + 	int negotiate_only = 0;
    ++	int porcelain = 0;
    + 	int i;
      
    -+static int opt_parse_porcelain(const struct option *opt, const char *arg, int unset)
    -+{
    -+	enum display_format *format = opt->value;
    -+	*format = DISPLAY_FORMAT_PORCELAIN;
    -+	return 0;
    -+}
    -+
    - int cmd_fetch(int argc, const char **argv, const char *prefix)
    - {
    - 	const char *bundle_uri;
    + 	struct option builtin_fetch_options[] = {
     @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
      		OPT_BOOL(0, "dry-run", &dry_run,
      			 N_("dry run")),
    -+		OPT_CALLBACK_F(0, "porcelain", &display_format, NULL, N_("machine-readable output"),
    -+			       PARSE_OPT_NOARG|PARSE_OPT_NONEG, opt_parse_porcelain),
    ++		OPT_BOOL(0, "porcelain", &porcelain, N_("machine-readable output")),
      		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
      			 N_("write fetched references to the FETCH_HEAD file")),
      		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
    @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      		fetch_config_from_gitmodules(sfjc, rs);
      	}
      
    -+	if (display_format == DISPLAY_FORMAT_PORCELAIN) {
    ++
    ++	if (porcelain) {
     +		switch (recurse_submodules_cli) {
     +		case RECURSE_SUBMODULES_OFF:
     +		case RECURSE_SUBMODULES_DEFAULT:
    @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
     +			die(_("options '%s' and '%s' cannot be used together"),
     +			    "--porcelain", "--recurse-submodules");
     +		}
    ++
    ++		config.display_format = DISPLAY_FORMAT_PORCELAIN;
     +	}
     +
      	if (negotiate_only && !negotiation_tip.nr)
    @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      
      		/* TODO should this also die if we have a previous partial-clone? */
     -		result = fetch_multiple(&list, max_children);
    -+		result = fetch_multiple(&list, max_children, display_format);
    ++		result = fetch_multiple(&list, max_children, config.display_format);
      	}
      
    - 
    +-
    + 	/*
    + 	 * This is only needed after fetch_one(), which does not fetch
    + 	 * submodules by itself.
     @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      		if (max_children < 0)
      			max_children = fetch_parallel_config;
      
     -		add_options_to_argv(&options);
    -+		add_options_to_argv(&options, display_format);
    ++		add_options_to_argv(&options, config.display_format);
      		result = fetch_submodules(the_repository,
      					  &options,
      					  submodule_prefix,
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch compact output' '
     +	test_must_be_empty stderr &&
     +	test_cmp expect stdout
     +'
    ++
    ++test_expect_success 'fetch --no-porcelain overrides previous --porcelain' '
    ++	test_when_finished "rm -rf no-porcelain" &&
    ++
    ++	git switch --create no-porcelain &&
    ++	git clone . no-porcelain &&
    ++	test_commit --no-tag no-porcelain &&
    ++	old_commit=$(git rev-parse --short HEAD~) &&
    ++	new_commit=$(git rev-parse --short HEAD) &&
    ++
    ++	cat >expect <<-EOF &&
    ++	From $(test-tool path-utils real_path .)/.
    ++	   $old_commit..$new_commit  no-porcelain -> origin/no-porcelain
    ++	EOF
    ++
    ++	git -C no-porcelain fetch --porcelain --no-porcelain >stdout 2>stderr &&
    ++	test_cmp expect stderr &&
    ++	test_must_be_empty stdout
    ++'
     +
      test_expect_success 'fetch output with HEAD' '
      	test_when_finished "rm -rf head" &&
    @@ t/t5574-fetch-output.sh: test_expect_success 'fetch output with HEAD' '
     +	test_cmp expect actual
     +'
     +
    - test_expect_success '--no-show-forced-updates' '
    - 	mkdir forced-updates &&
    - 	(
    + test_expect_success 'fetch output with object ID' '
    + 	test_when_finished "rm -rf object-id" &&
    + 	git clone . object-id &&
-- 
2.40.1
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Patrick Steinhardt · lore

[PATCH v5 1/9] fetch: fix `--no-recurse-submodules` with multi-remote fetches

When running `git fetch --no-recurse-submodules`, the exectation is that we don't fetch any submodules. And while this works for fetches of a single remote, it doesn't when fetching multiple remotes at once. The result is that we do recurse into submodules even though the user has explicitly asked us not to.

This is because while we pass on `--recurse-submodules={yes,on-demand}` if specified by the user, we don't pass on `--no-recurse-submodules` to the subprocess spawned to perform the submodule fetch.

Fix this by also forwarding this flag as expected.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c             |  2 ++
 t/t5526-fetch-submodules.sh | 13 +++++++++++++
 2 files changed, 15 insertions(+)
Show changes to 2 files +15 −0

builtin/fetch.c, t/t5526-fetch-submodules.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index c310d89878..08d7fc7233 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1876,6 +1876,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--keep");
 	if (recurse_submodules == RECURSE_SUBMODULES_ON)
 		strvec_push(argv, "--recurse-submodules");
+	else if (recurse_submodules == RECURSE_SUBMODULES_OFF)
+		strvec_push(argv, "--no-recurse-submodules");
 	else if (recurse_submodules == RECURSE_SUBMODULES_ON_DEMAND)
 		strvec_push(argv, "--recurse-submodules=on-demand");
 	if (tags == TAGS_SET)
diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
index dcdbe26a08..26e933f93a 100755
--- a/t/t5526-fetch-submodules.sh
+++ b/t/t5526-fetch-submodules.sh
@@ -1180,4 +1180,17 @@ test_expect_success 'fetch --all with --recurse-submodules with multiple' '
 	test_line_count = 2 fetch-subs
 '
 
+test_expect_success "fetch --all with --no-recurse-submodules only fetches superproject" '
+	test_when_finished "rm -rf src_clone" &&
+
+	git clone --recurse-submodules src src_clone &&
+	(
+		cd src_clone &&
+		git remote add secondary ../src &&
+		git config submodule.recurse true &&
+		git fetch --all --no-recurse-submodules 2>../fetch-log
+	) &&
+	! grep "Fetching submodule" fetch-log
+'
+
 test_done
-- 
2.40.1
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Patrick Steinhardt · lore

[PATCH v5 2/9] fetch: split out tests for output format

We're about to introduce a new porcelain mode for the output of git-fetch(1). As part of that we'll be introducing a set of new tests that only relate to the output of this command.

Split out tests that exercise the output format of git-fetch(1) so that it becomes easier to verify this functionality as a standalone unit. As the tests assume that the default branch is called "main" we set up the corresponding GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME environment variable accordingly.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5510-fetch.sh        | 53 ----------------------------------
 t/t5574-fetch-output.sh | 63 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+), 53 deletions(-)
 create mode 100755 t/t5574-fetch-output.sh
Show changes to 2 files +63 −53

t/t5510-fetch.sh, t/t5574-fetch-output.sh

diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index dc44da9c79..4f289063ce 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -1118,59 +1118,6 @@ test_expect_success 'fetching with auto-gc does not lock up' '
 	)
 '
 
-test_expect_success 'fetch aligned output' '
-	git clone . full-output &&
-	test_commit looooooooooooong-tag &&
-	(
-		cd full-output &&
-		git -c fetch.output=full fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main                 -> origin/main
-	looooooooooooong-tag -> looooooooooooong-tag
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success 'fetch compact output' '
-	git clone . compact &&
-	test_commit extraaa &&
-	(
-		cd compact &&
-		git -c fetch.output=compact fetch origin >actual 2>&1 &&
-		grep -e "->" actual | cut -c 22- >../actual
-	) &&
-	cat >expect <<-\EOF &&
-	main       -> origin/*
-	extraaa    -> *
-	EOF
-	test_cmp expect actual
-'
-
-test_expect_success '--no-show-forced-updates' '
-	mkdir forced-updates &&
-	(
-		cd forced-updates &&
-		git init &&
-		test_commit 1 &&
-		test_commit 2
-	) &&
-	git clone forced-updates forced-update-clone &&
-	git clone forced-updates no-forced-update-clone &&
-	git -C forced-updates reset --hard HEAD~1 &&
-	(
-		cd forced-update-clone &&
-		git fetch --show-forced-updates origin 2>output &&
-		test_i18ngrep "(forced update)" output
-	) &&
-	(
-		cd no-forced-update-clone &&
-		git fetch --no-show-forced-updates origin 2>output &&
-		test_i18ngrep ! "(forced update)" output
-	)
-'
-
 for section in fetch transfer
 do
 	test_expect_success "$section.hideRefs affects connectivity check" '
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
new file mode 100755
index 0000000000..f91b654d38
--- /dev/null
+++ b/t/t5574-fetch-output.sh
@@ -0,0 +1,63 @@
+#!/bin/sh
+
+test_description='git fetch output format'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+
+test_expect_success 'fetch aligned output' '
+	git clone . full-output &&
+	test_commit looooooooooooong-tag &&
+	(
+		cd full-output &&
+		git -c fetch.output=full fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main                 -> origin/main
+	looooooooooooong-tag -> looooooooooooong-tag
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success 'fetch compact output' '
+	git clone . compact &&
+	test_commit extraaa &&
+	(
+		cd compact &&
+		git -c fetch.output=compact fetch origin >actual 2>&1 &&
+		grep -e "->" actual | cut -c 22- >../actual
+	) &&
+	cat >expect <<-\EOF &&
+	main       -> origin/*
+	extraaa    -> *
+	EOF
+	test_cmp expect actual
+'
+
+test_expect_success '--no-show-forced-updates' '
+	mkdir forced-updates &&
+	(
+		cd forced-updates &&
+		git init &&
+		test_commit 1 &&
+		test_commit 2
+	) &&
+	git clone forced-updates forced-update-clone &&
+	git clone forced-updates no-forced-update-clone &&
+	git -C forced-updates reset --hard HEAD~1 &&
+	(
+		cd forced-update-clone &&
+		git fetch --show-forced-updates origin 2>output &&
+		test_i18ngrep "(forced update)" output
+	) &&
+	(
+		cd no-forced-update-clone &&
+		git fetch --no-show-forced-updates origin 2>output &&
+		test_i18ngrep ! "(forced update)" output
+	)
+'
+
+test_done
-- 
2.40.1
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Patrick Steinhardt · lore

[PATCH v5 3/9] fetch: add a test to exercise invalid output formats

Add a testcase that exercises the logic when an invalid output format is passed via the `fetch.output` configuration.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t5574-fetch-output.sh | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
Show changes to t/t5574-fetch-output.sh +24 −0
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index f91b654d38..8a344e6790 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -7,6 +7,30 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./test-lib.sh
 
+test_expect_success 'fetch with invalid output format configuration' '
+	test_when_finished "rm -rf clone" &&
+	git clone . clone &&
+
+	test_must_fail git -C clone -c fetch.output fetch origin 2>actual.err &&
+	cat >expect <<-EOF &&
+	error: missing value for ${SQ}fetch.output${SQ}
+	fatal: unable to parse ${SQ}fetch.output${SQ} from command-line config
+	EOF
+	test_cmp expect actual.err &&
+
+	test_must_fail git -C clone -c fetch.output= fetch origin 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}${SQ}
+	EOF
+	test_cmp expect actual.err &&
+
+	test_must_fail git -C clone -c fetch.output=garbage fetch origin 2>actual.err &&
+	cat >expect <<-EOF &&
+	fatal: invalid value for ${SQ}fetch.output${SQ}: ${SQ}garbage${SQ}
+	EOF
+	test_cmp expect actual.err
+'
+
 test_expect_success 'fetch aligned output' '
 	git clone . full-output &&
 	test_commit looooooooooooong-tag &&
-- 
2.40.1
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Patrick Steinhardt · lore

[PATCH v5 4/9] fetch: print left-hand side when fetching HEAD:foo

`store_updated_refs()` parses the remote reference for two purposes:
    - It gets used as a note when writing FETCH_HEAD.
    - It is passed through to `display_ref_update()` to display
      updated references in the following format:
      ```
       * branch               master          -> master
      ```

In most cases, the parsed remote reference is the prettified reference name and can thus be used for both cases. But if the remote reference is HEAD, the parsed remote reference becomes empty. This is intended when we write the FETCH_HEAD, where we skip writing the note in that case. But when displaying the updated references this leads to inconsistent output where the left-hand side of reference updates is missing in some cases:

```
$ git fetch origin HEAD HEAD:explicit-head :implicit-head main
From https://github.com/git/git
 * branch                  HEAD       -> FETCH_HEAD
 * [new ref]                          -> explicit-head
 * [new ref]                          -> implicit-head
 * branch                  main       -> FETCH_HEAD
```

This behaviour has existed ever since the table-based output has been introduced for git-fetch(1) via 165f390250 (git-fetch: more terse fetch output, 2007-11-03) and was never explicitly documented either in the commit message or in any of our tests. So while it may not be a bug per se, it feels like a weird inconsistency and not like it was a concious design decision.

The logic of how we compute the remote reference name that we ultimately pass to `display_ref_update()` is not easy to follow. There are three different cases here:

    - When the remote reference name is "HEAD" we set the remote
      reference name to the empty string. This is the case that causes
      the left-hand side to go missing, where we would indeed want to
      print "HEAD" instead of the empty string. This is what
      `prettify_refname()` would return.
    - When the remote reference name has a well-known prefix then we
      strip this prefix. This matches what `prettify_refname()` does.
    - Otherwise, we keep the fully qualified reference name. This also
      matches what `prettify_refname()` does.

As the return value of `prettify_refname()` would do the correct thing for us in all three cases, we can thus fix the inconsistency by passing through the full remote reference name to `display_ref_update()`, which learns to call `prettify_refname()`. At the same time, this also simplifies the code a bit.

Note that this patch also changes formatting of the block that computes the "kind" (which is the category like "branch" or "tag") and "what" (which is the prettified reference name like "master" or "v1.0") variables. This is done on purpose so that it is part of the diff, hopefully making the change easier to comprehend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c         | 37 ++++++++++++++++----------------
 t/t5574-fetch-output.sh | 47 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 18 deletions(-)
Show changes to 2 files +66 −18

builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 08d7fc7233..6aecf549e8 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -918,12 +918,14 @@ static void display_ref_update(struct display_state *display_state, char code,
 	}
 
 	width = (summary_width + strlen(summary) - gettext_width(summary));
+	remote = prettify_refname(remote);
+	local = prettify_refname(local);
 
 	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
 	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, prettify_refname(local));
+		print_remote_to_local(display_state, remote, local);
 	else
-		print_compact(display_state, remote, prettify_refname(local));
+		print_compact(display_state, remote, local);
 	if (error)
 		strbuf_addf(&display_state->buf, "  (%s)", error);
 	strbuf_addch(&display_state->buf, '\n');
@@ -934,7 +936,7 @@ static void display_ref_update(struct display_state *display_state, char code,
 static int update_local_ref(struct ref *ref,
 			    struct ref_transaction *transaction,
 			    struct display_state *display_state,
-			    const char *remote, const struct ref *remote_ref,
+			    const struct ref *remote_ref,
 			    int summary_width)
 {
 	struct commit *current = NULL, *updated;
@@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 		return 0;
 	}
 
@@ -959,7 +961,7 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 
@@ -970,12 +972,12 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote, ref->name, summary_width);
+					   remote_ref->name, ref->name, summary_width);
 			return 1;
 		}
 	}
@@ -1008,7 +1010,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return r;
 	}
 
@@ -1030,7 +1032,7 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -1042,12 +1044,12 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote, ref->name, summary_width);
+				   remote_ref->name, ref->name, summary_width);
 		return 1;
 	}
 }
@@ -1252,14 +1254,13 @@ static int store_updated_refs(struct display_state *display_state,
 			if (!strcmp(rm->name, "HEAD")) {
 				kind = "";
 				what = "";
-			}
-			else if (skip_prefix(rm->name, "refs/heads/", &what))
+			} else if (skip_prefix(rm->name, "refs/heads/", &what)) {
 				kind = "branch";
-			else if (skip_prefix(rm->name, "refs/tags/", &what))
+			} else if (skip_prefix(rm->name, "refs/tags/", &what)) {
 				kind = "tag";
-			else if (skip_prefix(rm->name, "refs/remotes/", &what))
+			} else if (skip_prefix(rm->name, "refs/remotes/", &what)) {
 				kind = "remote-tracking branch";
-			else {
+			} else {
 				kind = "";
 				what = rm->name;
 			}
@@ -1277,7 +1278,7 @@ static int store_updated_refs(struct display_state *display_state,
 					  display_state->url_len);
 
 			if (ref) {
-				rc |= update_local_ref(ref, transaction, display_state, what,
+				rc |= update_local_ref(ref, transaction, display_state,
 						       rm, summary_width);
 				free(ref);
 			} else if (write_fetch_head || dry_run) {
@@ -1288,7 +1289,7 @@ static int store_updated_refs(struct display_state *display_state,
 				 */
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
-						   *what ? what : "HEAD",
+						   rm->name,
 						   "FETCH_HEAD", summary_width);
 			}
 		}
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index 8a344e6790..9890f6f381 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -61,6 +61,53 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch output with HEAD' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+
+	git -C head fetch --dry-run origin HEAD >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * branch            HEAD       -> FETCH_HEAD
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch --dry-run origin HEAD:foo >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         HEAD       -> foo
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C head fetch origin HEAD:foo >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
+test_expect_success 'fetch output with object ID' '
+	test_when_finished "rm -rf object-id" &&
+	git clone . object-id &&
+	commit=$(git rev-parse HEAD) &&
+
+	git -C object-id fetch --dry-run origin $commit:object-id >actual.out 2>actual.err &&
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	 * [new ref]         $commit -> object-id
+	EOF
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err &&
+
+	git -C object-id fetch origin $commit:object-id >actual.out 2>actual.err &&
+	test_must_be_empty actual.out &&
+	test_cmp expect actual.err
+'
+
 test_expect_success '--no-show-forced-updates' '
 	mkdir forced-updates &&
 	(
-- 
2.40.1
Glen Choo· May 12, 2023, 00:16 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v5 4/9] fetch: print left-hand side when fetching HEAD:foo

Patrick Steinhardt <ps@pks.im> writes:
Show 6 quoted lines
> This behaviour has existed ever since the table-based output has been
> introduced for git-fetch(1) via 165f390250 (git-fetch: more terse fetch
> output, 2007-11-03) and was never explicitly documented either in the
> commit message or in any of our tests. So while it may not be a bug per
> se, it feels like a weird inconsistency and not like it was a concious
> design decision.

The change seems well-justified. I agree that it does feel strange and unintended. I was really surprised to learn that it was not a bug.

Jeff King· May 13, 2023, 16:59 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v5 4/9] fetch: print left-hand side when fetching HEAD:foo

On Wed, May 10, 2023 at 02:34:15PM +0200, Patrick Steinhardt wrote:
Show 8 quoted lines
> @@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
>  	if (oideq(&ref->old_oid, &ref->new_oid)) {
>  		if (verbosity > 0)
>  			display_ref_update(display_state, '=', _("[up to date]"), NULL,
> -					   remote, ref->name, summary_width);
> +					   remote_ref->name, ref->name, summary_width);
>  		return 0;
>  	}

Here (and in other hunks) we now dereference remote_ref unconditionally. But in existing parts of the code, we guard against remote_ref being NULL. E.g., later on:

          if (!current || !updated) {
                  const char *msg;
                  const char *what;
                  int r;
                  /*
                   * Nicely describe the new ref we're fetching.
                   * Base this on the remote's ref name, as it's
                   * more likely to follow a standard layout.
                   */
                  const char *name = remote_ref ? remote_ref->name : "";
		  [...]

I'm not sure if the old code was being overly defensive, or if the new code is ripe for a segfault. But it's probably worth looking into (it was noticed by Coverity).

Looking at the caller, it is always store_update_refs() which passes its own "rm", a pointer iterating over ref_map. And it should always be non-NULL, since that's the loop condition.

So I think your code is fine, but you might want to double-check my logic. (And it may be worth cleaning up the existing redundant check to prevent confusion).

-Peff
Patrick Steinhardt· May 15, 2023, 05:15 UTC · re: Jeff King · lore

Re: [PATCH v5 4/9] fetch: print left-hand side when fetching HEAD:foo

On Sat, May 13, 2023 at 12:59:25PM -0400, Jeff King wrote:
Show 38 quoted lines
> On Wed, May 10, 2023 at 02:34:15PM +0200, Patrick Steinhardt wrote:
> 
> > @@ -946,7 +948,7 @@ static int update_local_ref(struct ref *ref,
> >  	if (oideq(&ref->old_oid, &ref->new_oid)) {
> >  		if (verbosity > 0)
> >  			display_ref_update(display_state, '=', _("[up to date]"), NULL,
> > -					   remote, ref->name, summary_width);
> > +					   remote_ref->name, ref->name, summary_width);
> >  		return 0;
> >  	}
> 
> Here (and in other hunks) we now dereference remote_ref unconditionally.
> But in existing parts of the code, we guard against remote_ref being
> NULL. E.g., later on:
> 
>           if (!current || !updated) {
>                   const char *msg;
>                   const char *what;
>                   int r;
>                   /*
>                    * Nicely describe the new ref we're fetching.
>                    * Base this on the remote's ref name, as it's
>                    * more likely to follow a standard layout.
>                    */
>                   const char *name = remote_ref ? remote_ref->name : "";
> 		  [...]
> 
> I'm not sure if the old code was being overly defensive, or if the new
> code is ripe for a segfault. But it's probably worth looking into (it
> was noticed by Coverity).
> 
> Looking at the caller, it is always store_update_refs() which passes its
> own "rm", a pointer iterating over ref_map. And it should always be
> non-NULL, since that's the loop condition.
> 
> So I think your code is fine, but you might want to double-check my
> logic. (And it may be worth cleaning up the existing redundant check to
> prevent confusion).

I really think that the code is overly defensive. As you mention, there is a single caller of `update_local_ref()`, only, and that caller already dereferences the remote reference unconditionally anyway. So if there was any way for `rm` to become `NULL` we'd already have a segfault earlier than in `update_local_ref()`.

I'll send a follow-up patch series after the dust has settled that removes the check.

Thanks!
Patrick
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Patrick Steinhardt · lore

[PATCH v5 5/9] fetch: refactor calculation of the display table width

When displaying reference updates, we try to print the references in a neat table. As the table's width is determined its contents we thus need to precalculate the overall width before we can start printing updated references.

The calculation is driven by `display_state_init()`, which invokes `refcol_width()` for every reference that is to be printed. This split is somewhat confusing. For one, we filter references that shall be attributed to the overall width in both places. And second, we needlessly recalculate the maximum line length based on the terminal columns and display format for every reference.

Refactor the code so that the complete width calculations are neatly contained in `refcol_width()`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 81 ++++++++++++++++++++++---------------------------
 1 file changed, 37 insertions(+), 44 deletions(-)
Show changes to builtin/fetch.c +37 −44
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 6aecf549e8..007eb3693d 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -751,40 +751,51 @@ static int s_update_ref(const char *action,
 	return ret;
 }
 
-static int refcol_width(const struct ref *ref, int compact_format)
+static int refcol_width(const struct ref *ref_map, int compact_format)
 {
-	int max, rlen, llen, len;
+	const struct ref *ref;
+	int max, width = 10;
 
-	/* uptodate lines are only shown on high verbosity level */
-	if (verbosity <= 0 && oideq(&ref->peer_ref->old_oid, &ref->old_oid))
-		return 0;
-
-	max    = term_columns();
-	rlen   = utf8_strwidth(prettify_refname(ref->name));
-
-	llen   = utf8_strwidth(prettify_refname(ref->peer_ref->name));
-
-	/*
-	 * rough estimation to see if the output line is too long and
-	 * should not be counted (we can't do precise calculation
-	 * anyway because we don't know if the error explanation part
-	 * will be printed in update_local_ref)
-	 */
-	if (compact_format) {
-		llen = 0;
+	max = term_columns();
+	if (compact_format)
 		max = max * 2 / 3;
-	}
-	len = 21 /* flag and summary */ + rlen + 4 /* -> */ + llen;
-	if (len >= max)
-		return 0;
 
-	return rlen;
+	for (ref = ref_map; ref; ref = ref->next) {
+		int rlen, llen = 0, len;
+
+		if (ref->status == REF_STATUS_REJECT_SHALLOW ||
+		    !ref->peer_ref ||
+		    !strcmp(ref->name, "HEAD"))
+			continue;
+
+		/* uptodate lines are only shown on high verbosity level */
+		if (verbosity <= 0 && oideq(&ref->peer_ref->old_oid, &ref->old_oid))
+			continue;
+
+		rlen = utf8_strwidth(prettify_refname(ref->name));
+		if (!compact_format)
+			llen = utf8_strwidth(prettify_refname(ref->peer_ref->name));
+
+		/*
+		 * rough estimation to see if the output line is too long and
+		 * should not be counted (we can't do precise calculation
+		 * anyway because we don't know if the error explanation part
+		 * will be printed in update_local_ref)
+		 */
+		len = 21 /* flag and summary */ + rlen + 4 /* -> */ + llen;
+		if (len >= max)
+			continue;
+
+		if (width < rlen)
+			width = rlen;
+	}
+
+	return width;
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
 			       const char *raw_url)
 {
-	struct ref *rm;
 	const char *format = "full";
 	int i;
 
@@ -816,25 +827,7 @@ static void display_state_init(struct display_state *display_state, struct ref *
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = 10;
-	for (rm = ref_map; rm; rm = rm->next) {
-		int width;
-
-		if (rm->status == REF_STATUS_REJECT_SHALLOW ||
-		    !rm->peer_ref ||
-		    !strcmp(rm->name, "HEAD"))
-			continue;
-
-		width = refcol_width(rm, display_state->compact_format);
-
-		/*
-		 * Not precise calculation for compact mode because '*' can
-		 * appear on the left hand side of '->' and shrink the column
-		 * back.
-		 */
-		if (display_state->refcol_width < width)
-			display_state->refcol_width = width;
-	}
+	display_state->refcol_width = refcol_width(ref_map, display_state->compact_format);
 }
 
 static void display_state_release(struct display_state *display_state)
-- 
2.40.1
Glen Choo· May 12, 2023, 00:49 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v5 5/9] fetch: refactor calculation of the display table width

Patrick Steinhardt <ps@pks.im> writes:
Show 14 quoted lines
> When displaying reference updates, we try to print the references in a
> neat table. As the table's width is determined its contents we thus need
> to precalculate the overall width before we can start printing updated
> references.
>
> The calculation is driven by `display_state_init()`, which invokes
> `refcol_width()` for every reference that is to be printed. This split
> is somewhat confusing. For one, we filter references that shall be
> attributed to the overall width in both places. And second, we
> needlessly recalculate the maximum line length based on the terminal
> columns and display format for every reference.
>
> Refactor the code so that the complete width calculations are neatly
> contained in `refcol_width()`.

Through no fault of yours, I have to admit that I found this refactor quite hard to read. I ended up redoing the refactor and ended up with a result very similar to yours. That was probably overkill since we have pretty extensive tests in this area, but I'm quite happy with the change since it's far more readable.

Show 15 quoted lines
> -	/*
> -	 * rough estimation to see if the output line is too long and
> -	 * should not be counted (we can't do precise calculation
> -	 * anyway because we don't know if the error explanation part
> -	 * will be printed in update_local_ref)
> -	 */
> -	if (compact_format) {
> -		llen = 0;
> +	max = term_columns();
> +	if (compact_format)
>  		max = max * 2 / 3;
> -	}
> -	len = 21 /* flag and summary */ + rlen + 4 /* -> */ + llen;
> -	if (len >= max)
> -		return 0;

The one thing that changed for the better (vs keeping the lines in the same order as before) is that this comment that used to be anchored on "if (compact_format) {"...

Show 9 quoted lines
> +		/*
> +		 * rough estimation to see if the output line is too long and
> +		 * should not be counted (we can't do precise calculation
> +		 * anyway because we don't know if the error explanation part
> +		 * will be printed in update_local_ref)
> +		 */
> +		len = 21 /* flag and summary */ + rlen + 4 /* -> */ + llen;
> +		if (len >= max)
> +			continue;
is now more accurately anchored to the check that throws away the line.
Looks good.
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Patrick Steinhardt · lore

[PATCH v5 6/9] fetch: introduce `display_format` enum

We currently have two different display formats in git-fetch(1) with the "full" and "compact" formats. This is tracked with a boolean value that simply denotes whether the display format is supposed to be compacted or not. This works reasonably well while there are only two formats, but we're about to introduce another format that will make this a bit more awkward to use.

Introduce a `enum display_format` that is more readily extensible.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 69 +++++++++++++++++++++++++++++++++----------------
 1 file changed, 47 insertions(+), 22 deletions(-)
Show changes to builtin/fetch.c +47 −22
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 007eb3693d..4c58ee503a 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -48,11 +48,17 @@ enum {
 	TAGS_SET = 2
 };
 
+enum display_format {
+	DISPLAY_FORMAT_UNKNOWN = 0,
+	DISPLAY_FORMAT_FULL,
+	DISPLAY_FORMAT_COMPACT,
+};
+
 struct display_state {
 	struct strbuf buf;
 
 	int refcol_width;
-	int compact_format;
+	enum display_format format;
 
 	char *url;
 	int url_len, shown_url;
@@ -820,14 +826,22 @@ static void display_state_init(struct display_state *display_state, struct ref *
 
 	git_config_get_string_tmp("fetch.output", &format);
 	if (!strcasecmp(format, "full"))
-		display_state->compact_format = 0;
+		display_state->format = DISPLAY_FORMAT_FULL;
 	else if (!strcasecmp(format, "compact"))
-		display_state->compact_format = 1;
+		display_state->format = DISPLAY_FORMAT_COMPACT;
 	else
 		die(_("invalid value for '%s': '%s'"),
 		    "fetch.output", format);
 
-	display_state->refcol_width = refcol_width(ref_map, display_state->compact_format);
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT:
+		display_state->refcol_width = refcol_width(ref_map,
+							   display_state->format == DISPLAY_FORMAT_COMPACT);
+		break;
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	}
 }
 
 static void display_state_release(struct display_state *display_state)
@@ -897,30 +911,41 @@ static void display_ref_update(struct display_state *display_state, char code,
 			       const char *remote, const char *local,
 			       int summary_width)
 {
-	int width;
-
 	if (verbosity < 0)
 		return;
 
 	strbuf_reset(&display_state->buf);
 
-	if (!display_state->shown_url) {
-		strbuf_addf(&display_state->buf, _("From %.*s\n"),
-			    display_state->url_len, display_state->url);
-		display_state->shown_url = 1;
+	switch (display_state->format) {
+	case DISPLAY_FORMAT_FULL:
+	case DISPLAY_FORMAT_COMPACT: {
+		int width;
+
+		if (!display_state->shown_url) {
+			strbuf_addf(&display_state->buf, _("From %.*s\n"),
+				    display_state->url_len, display_state->url);
+			display_state->shown_url = 1;
+		}
+
+		width = (summary_width + strlen(summary) - gettext_width(summary));
+		remote = prettify_refname(remote);
+		local = prettify_refname(local);
+
+		strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
+
+		if (display_state->format != DISPLAY_FORMAT_COMPACT)
+			print_remote_to_local(display_state, remote, local);
+		else
+			print_compact(display_state, remote, local);
+
+		if (error)
+			strbuf_addf(&display_state->buf, "  (%s)", error);
+
+		break;
 	}
-
-	width = (summary_width + strlen(summary) - gettext_width(summary));
-	remote = prettify_refname(remote);
-	local = prettify_refname(local);
-
-	strbuf_addf(&display_state->buf, " %c %-*s ", code, width, summary);
-	if (!display_state->compact_format)
-		print_remote_to_local(display_state, remote, local);
-	else
-		print_compact(display_state, remote, local);
-	if (error)
-		strbuf_addf(&display_state->buf, "  (%s)", error);
+	default:
+		BUG("unexpected display format %d", display_state->format);
+	};
 	strbuf_addch(&display_state->buf, '\n');
 
 	fputs(display_state->buf.buf, stderr);
-- 
2.40.1
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Patrick Steinhardt · lore

[PATCH v5 7/9] fetch: lift up parsing of "fetch.output" config variable

Parsing the display format happens inside of `display_state_init()`. As we only need to check for a simple config entry, this is a natural location to put this code as it means that display-state logic is neatly contained in a single location.

We're about to introduce a new "porcelain" output format though that is intended to be parseable by machines, for example inside of a script. This format can be enabled by passing the `--porcelain` switch to git-fetch(1). As a consequence, we'll have to add a second callsite that influences the output format, which will become awkward to handle.

Refactor the code such that callers are expected to pass the display format that is to be used into `display_state_init()`. This allows us to lift up the code into the main function, where we can then hook it into command line options parser in a follow-up commit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 50 +++++++++++++++++++++++++++++++------------------
 1 file changed, 32 insertions(+), 18 deletions(-)
Show changes to builtin/fetch.c +32 −18
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 4c58ee503a..439d3c667b 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -104,8 +104,14 @@ static int fetch_write_commit_graph = -1;
 static int stdin_refspecs = 0;
 static int negotiate_only;
 
+struct fetch_config {
+	enum display_format display_format;
+};
+
 static int git_fetch_config(const char *k, const char *v, void *cb)
 {
+	struct fetch_config *fetch_config = cb;
+
 	if (!strcmp(k, "fetch.prune")) {
 		fetch_prune_config = git_config_bool(k, v);
 		return 0;
@@ -144,6 +150,18 @@ static int git_fetch_config(const char *k, const char *v, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(k, "fetch.output")) {
+		if (!v)
+			return config_error_nonbool(k);
+		else if (!strcasecmp(v, "full"))
+			fetch_config->display_format = DISPLAY_FORMAT_FULL;
+		else if (!strcasecmp(v, "compact"))
+			fetch_config->display_format = DISPLAY_FORMAT_COMPACT;
+		else
+			die(_("invalid value for '%s': '%s'"),
+			    "fetch.output", v);
+	}
+
 	return git_default_config(k, v, cb);
 }
 
@@ -800,14 +818,13 @@ static int refcol_width(const struct ref *ref_map, int compact_format)
 }
 
 static void display_state_init(struct display_state *display_state, struct ref *ref_map,
-			       const char *raw_url)
+			       const char *raw_url, enum display_format format)
 {
-	const char *format = "full";
 	int i;
 
 	memset(display_state, 0, sizeof(*display_state));
-
 	strbuf_init(&display_state->buf, 0);
+	display_state->format = format;
 
 	if (raw_url)
 		display_state->url = transport_anonymize_url(raw_url);
@@ -824,15 +841,6 @@ static void display_state_init(struct display_state *display_state, struct ref *
 	if (verbosity < 0)
 		return;
 
-	git_config_get_string_tmp("fetch.output", &format);
-	if (!strcasecmp(format, "full"))
-		display_state->format = DISPLAY_FORMAT_FULL;
-	else if (!strcasecmp(format, "compact"))
-		display_state->format = DISPLAY_FORMAT_COMPACT;
-	else
-		die(_("invalid value for '%s': '%s'"),
-		    "fetch.output", format);
-
 	switch (display_state->format) {
 	case DISPLAY_FORMAT_FULL:
 	case DISPLAY_FORMAT_COMPACT:
@@ -1605,7 +1613,8 @@ static int backfill_tags(struct display_state *display_state,
 }
 
 static int do_fetch(struct transport *transport,
-		    struct refspec *rs)
+		    struct refspec *rs,
+		    enum display_format display_format)
 {
 	struct ref_transaction *transaction = NULL;
 	struct ref *ref_map = NULL;
@@ -1691,7 +1700,7 @@ static int do_fetch(struct transport *transport,
 	if (retcode)
 		goto cleanup;
 
-	display_state_init(&display_state, ref_map, transport->url);
+	display_state_init(&display_state, ref_map, transport->url, display_format);
 
 	if (atomic_fetch) {
 		transaction = ref_transaction_begin(&err);
@@ -2069,7 +2078,8 @@ static inline void fetch_one_setup_partial(struct remote *remote)
 }
 
 static int fetch_one(struct remote *remote, int argc, const char **argv,
-		     int prune_tags_ok, int use_stdin_refspecs)
+		     int prune_tags_ok, int use_stdin_refspecs,
+		     enum display_format display_format)
 {
 	struct refspec rs = REFSPEC_INIT_FETCH;
 	int i;
@@ -2136,7 +2146,7 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 	sigchain_push_common(unlock_pack_on_signal);
 	atexit(unlock_pack_atexit);
 	sigchain_push(SIGPIPE, SIG_IGN);
-	exit_code = do_fetch(gtransport, &rs);
+	exit_code = do_fetch(gtransport, &rs, display_format);
 	sigchain_pop(SIGPIPE);
 	refspec_clear(&rs);
 	transport_disconnect(gtransport);
@@ -2146,6 +2156,9 @@ static int fetch_one(struct remote *remote, int argc, const char **argv,
 
 int cmd_fetch(int argc, const char **argv, const char *prefix)
 {
+	struct fetch_config config = {
+		.display_format = DISPLAY_FORMAT_FULL,
+	};
 	int i;
 	const char *bundle_uri;
 	struct string_list list = STRING_LIST_INIT_DUP;
@@ -2165,7 +2178,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		free(anon);
 	}
 
-	git_config(git_fetch_config, NULL);
+	git_config(git_fetch_config, &config);
 	if (the_repository->gitdir) {
 		prepare_repo_settings(the_repository);
 		the_repository->settings.command_requires_full_index = 0;
@@ -2302,7 +2315,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	} else if (remote) {
 		if (filter_options.choice || has_promisor_remote())
 			fetch_one_setup_partial(remote);
-		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs);
+		result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs,
+				   config.display_format);
 	} else {
 		int max_children = max_jobs;
 
-- 
2.40.1
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Patrick Steinhardt · lore

[PATCH v5 8/9] fetch: move option related variables into main function

The options of git-fetch(1) which we pass to `parse_options()` are declared globally in `builtin/fetch.c`. This means we're forced to use global variables for all the options, which is more likely to cause confusion than explicitly passing state around.

Refactor the code to move the options into `cmd_fetch()`. Move variables that were previously forced to be declared globally and which are only used by `cmd_fetch()` into function-local scope.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/fetch.c | 197 ++++++++++++++++++++++++------------------------
 1 file changed, 100 insertions(+), 97 deletions(-)
Show changes to builtin/fetch.c +100 −97
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 439d3c667b..7a3d620c4c 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -75,13 +75,12 @@ static int fetch_prune_tags_config = -1; /* unspecified */
 static int prune_tags = -1; /* unspecified */
 #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */
 
-static int all, append, dry_run, force, keep, multiple, update_head_ok;
+static int append, dry_run, force, keep, update_head_ok;
 static int write_fetch_head = 1;
 static int verbosity, deepen_relative, set_upstream, refetch;
 static int progress = -1;
-static int enable_auto_gc = 1;
-static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
-static int max_jobs = -1, submodule_fetch_jobs_config = -1;
+static int tags = TAGS_DEFAULT, update_shallow, deepen;
+static int submodule_fetch_jobs_config = -1;
 static int fetch_parallel_config = 1;
 static int atomic_fetch;
 static enum transport_family family;
@@ -92,17 +91,11 @@ static struct string_list deepen_not = STRING_LIST_INIT_NODUP;
 static struct strbuf default_rla = STRBUF_INIT;
 static struct transport *gtransport;
 static struct transport *gsecondary;
-static const char *submodule_prefix = "";
 static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
-static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
 static struct refspec refmap = REFSPEC_INIT_FETCH;
 static struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT;
 static struct string_list server_options = STRING_LIST_INIT_DUP;
 static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;
-static int fetch_write_commit_graph = -1;
-static int stdin_refspecs = 0;
-static int negotiate_only;
 
 struct fetch_config {
 	enum display_format display_format;
@@ -178,92 +171,6 @@ static int parse_refmap_arg(const struct option *opt, const char *arg, int unset
 	return 0;
 }
 
-static struct option builtin_fetch_options[] = {
-	OPT__VERBOSITY(&verbosity),
-	OPT_BOOL(0, "all", &all,
-		 N_("fetch from all remotes")),
-	OPT_BOOL(0, "set-upstream", &set_upstream,
-		 N_("set upstream for git pull/fetch")),
-	OPT_BOOL('a', "append", &append,
-		 N_("append to .git/FETCH_HEAD instead of overwriting")),
-	OPT_BOOL(0, "atomic", &atomic_fetch,
-		 N_("use atomic transaction to update references")),
-	OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
-		   N_("path to upload pack on remote end")),
-	OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
-	OPT_BOOL('m', "multiple", &multiple,
-		 N_("fetch from multiple remotes")),
-	OPT_SET_INT('t', "tags", &tags,
-		    N_("fetch all tags and associated objects"), TAGS_SET),
-	OPT_SET_INT('n', NULL, &tags,
-		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
-	OPT_INTEGER('j', "jobs", &max_jobs,
-		    N_("number of submodules fetched in parallel")),
-	OPT_BOOL(0, "prefetch", &prefetch,
-		 N_("modify the refspec to place all refs within refs/prefetch/")),
-	OPT_BOOL('p', "prune", &prune,
-		 N_("prune remote-tracking branches no longer on remote")),
-	OPT_BOOL('P', "prune-tags", &prune_tags,
-		 N_("prune local tags no longer on remote and clobber changed tags")),
-	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
-		    N_("control recursive fetching of submodules"),
-		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "dry-run", &dry_run,
-		 N_("dry run")),
-	OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
-		 N_("write fetched references to the FETCH_HEAD file")),
-	OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
-	OPT_BOOL('u', "update-head-ok", &update_head_ok,
-		    N_("allow updating of HEAD ref")),
-	OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
-	OPT_STRING(0, "depth", &depth, N_("depth"),
-		   N_("deepen history of shallow clone")),
-	OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
-		   N_("deepen history of shallow repository based on time")),
-	OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
-			N_("deepen history of shallow clone, excluding rev")),
-	OPT_INTEGER(0, "deepen", &deepen_relative,
-		    N_("deepen history of shallow clone")),
-	OPT_SET_INT_F(0, "unshallow", &unshallow,
-		      N_("convert to a complete repository"),
-		      1, PARSE_OPT_NONEG),
-	OPT_SET_INT_F(0, "refetch", &refetch,
-		      N_("re-fetch without negotiating common commits"),
-		      1, PARSE_OPT_NONEG),
-	{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
-		   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
-	OPT_CALLBACK_F(0, "recurse-submodules-default",
-		   &recurse_submodules_default, N_("on-demand"),
-		   N_("default for recursive fetching of submodules "
-		      "(lower priority than config files)"),
-		   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
-	OPT_BOOL(0, "update-shallow", &update_shallow,
-		 N_("accept refs that update .git/shallow")),
-	OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
-		       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
-	OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
-	OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
-			TRANSPORT_FAMILY_IPV4),
-	OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
-			TRANSPORT_FAMILY_IPV6),
-	OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
-			N_("report that we have only objects reachable from this object")),
-	OPT_BOOL(0, "negotiate-only", &negotiate_only,
-		 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
-	OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
-	OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "auto-gc", &enable_auto_gc,
-		 N_("run 'maintenance --auto' after fetching")),
-	OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
-		 N_("check for forced-updates on all updated branches")),
-	OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
-		 N_("write the commit-graph after fetching")),
-	OPT_BOOL(0, "stdin", &stdin_refspecs,
-		 N_("accept refspecs from stdin")),
-	OPT_END()
-};
-
 static void unlock_pack(unsigned int flags)
 {
 	if (gtransport)
@@ -2159,12 +2066,108 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	struct fetch_config config = {
 		.display_format = DISPLAY_FORMAT_FULL,
 	};
-	int i;
+	const char *submodule_prefix = "";
 	const char *bundle_uri;
 	struct string_list list = STRING_LIST_INIT_DUP;
 	struct remote *remote = NULL;
+	int all = 0, multiple = 0;
 	int result = 0;
 	int prune_tags_ok = 1;
+	int enable_auto_gc = 1;
+	int unshallow = 0;
+	int max_jobs = -1;
+	int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
+	int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
+	int fetch_write_commit_graph = -1;
+	int stdin_refspecs = 0;
+	int negotiate_only = 0;
+	int i;
+
+	struct option builtin_fetch_options[] = {
+		OPT__VERBOSITY(&verbosity),
+		OPT_BOOL(0, "all", &all,
+			 N_("fetch from all remotes")),
+		OPT_BOOL(0, "set-upstream", &set_upstream,
+			 N_("set upstream for git pull/fetch")),
+		OPT_BOOL('a', "append", &append,
+			 N_("append to .git/FETCH_HEAD instead of overwriting")),
+		OPT_BOOL(0, "atomic", &atomic_fetch,
+			 N_("use atomic transaction to update references")),
+		OPT_STRING(0, "upload-pack", &upload_pack, N_("path"),
+			   N_("path to upload pack on remote end")),
+		OPT__FORCE(&force, N_("force overwrite of local reference"), 0),
+		OPT_BOOL('m', "multiple", &multiple,
+			 N_("fetch from multiple remotes")),
+		OPT_SET_INT('t', "tags", &tags,
+			    N_("fetch all tags and associated objects"), TAGS_SET),
+		OPT_SET_INT('n', NULL, &tags,
+			    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
+		OPT_INTEGER('j', "jobs", &max_jobs,
+			    N_("number of submodules fetched in parallel")),
+		OPT_BOOL(0, "prefetch", &prefetch,
+			 N_("modify the refspec to place all refs within refs/prefetch/")),
+		OPT_BOOL('p', "prune", &prune,
+			 N_("prune remote-tracking branches no longer on remote")),
+		OPT_BOOL('P', "prune-tags", &prune_tags,
+			 N_("prune local tags no longer on remote and clobber changed tags")),
+		OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
+			    N_("control recursive fetching of submodules"),
+			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "dry-run", &dry_run,
+			 N_("dry run")),
+		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
+			 N_("write fetched references to the FETCH_HEAD file")),
+		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
+		OPT_BOOL('u', "update-head-ok", &update_head_ok,
+			    N_("allow updating of HEAD ref")),
+		OPT_BOOL(0, "progress", &progress, N_("force progress reporting")),
+		OPT_STRING(0, "depth", &depth, N_("depth"),
+			   N_("deepen history of shallow clone")),
+		OPT_STRING(0, "shallow-since", &deepen_since, N_("time"),
+			   N_("deepen history of shallow repository based on time")),
+		OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
+				N_("deepen history of shallow clone, excluding rev")),
+		OPT_INTEGER(0, "deepen", &deepen_relative,
+			    N_("deepen history of shallow clone")),
+		OPT_SET_INT_F(0, "unshallow", &unshallow,
+			      N_("convert to a complete repository"),
+			      1, PARSE_OPT_NONEG),
+		OPT_SET_INT_F(0, "refetch", &refetch,
+			      N_("re-fetch without negotiating common commits"),
+			      1, PARSE_OPT_NONEG),
+		{ OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"),
+			   N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN },
+		OPT_CALLBACK_F(0, "recurse-submodules-default",
+			   &recurse_submodules_default, N_("on-demand"),
+			   N_("default for recursive fetching of submodules "
+			      "(lower priority than config files)"),
+			   PARSE_OPT_HIDDEN, option_fetch_parse_recurse_submodules),
+		OPT_BOOL(0, "update-shallow", &update_shallow,
+			 N_("accept refs that update .git/shallow")),
+		OPT_CALLBACK_F(0, "refmap", NULL, N_("refmap"),
+			       N_("specify fetch refmap"), PARSE_OPT_NONEG, parse_refmap_arg),
+		OPT_STRING_LIST('o', "server-option", &server_options, N_("server-specific"), N_("option to transmit")),
+		OPT_SET_INT('4', "ipv4", &family, N_("use IPv4 addresses only"),
+				TRANSPORT_FAMILY_IPV4),
+		OPT_SET_INT('6', "ipv6", &family, N_("use IPv6 addresses only"),
+				TRANSPORT_FAMILY_IPV6),
+		OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"),
+				N_("report that we have only objects reachable from this object")),
+		OPT_BOOL(0, "negotiate-only", &negotiate_only,
+			 N_("do not fetch a packfile; instead, print ancestors of negotiation tips")),
+		OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
+		OPT_BOOL(0, "auto-maintenance", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "auto-gc", &enable_auto_gc,
+			 N_("run 'maintenance --auto' after fetching")),
+		OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates,
+			 N_("check for forced-updates on all updated branches")),
+		OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph,
+			 N_("write the commit-graph after fetching")),
+		OPT_BOOL(0, "stdin", &stdin_refspecs,
+			 N_("accept refspecs from stdin")),
+		OPT_END()
+	};
 
 	packet_trace_identity("fetch");
 
-- 
2.40.1
Patrick Steinhardt· May 10, 2023, 12:34 UTC · re: Patrick Steinhardt · lore

[PATCH v5 9/9] fetch: introduce machine-parseable "porcelain" output format

The output of git-fetch(1) is obviously designed for consumption by users, only: we neatly columnize data, we abbreviate reference names, we print neat arrows and we don't provide information about actual object IDs that have changed. This makes the output format basically unusable in the context of scripted invocations of git-fetch(1) that want to learn about the exact changes that the command performs.

Introduce a new machine-parseable "porcelain" output format that is supposed to fix this shortcoming. This output format is intended to provide information about every reference that is about to be updated, the old object ID that the reference has been pointing to and the new object ID it will be updated to. Furthermore, the output format provides the same flags as the human-readable format to indicate basic conditions for each reference update like whether it was a fast-forward update, a branch deletion, a rejected update or others.

The output format is quite simple:

``` <flag> <old-object-id> <new-object-id> <local-reference>\n ```

We assume two conditions which are generally true:
    - The old and new object IDs have fixed known widths and cannot
      contain spaces.
    - References cannot contain newlines.

With these assumptions, the output format becomes unambiguously parseable. Furthermore, given that this output is designed to be consumed by scripts, the machine-readable data is printed to stdout instead of stderr like the human-readable output is. This is mostly done so that other data printed to stderr, like error messages or progress meters, don't interfere with the parseable data.

A notable ommission here is that the output format does not include the remote from which a reference was fetched, which might be important information especially in the context of multi-remote fetches. But as such a format would require us to print the remote for every single reference update due to parallelizable fetches it feels wasteful for the most likely usecase, which is when fetching from a single remote.

In a similar spirit, a second restriction is that this cannot be used with `--recurse-submodules`. This is because any reference updates would be ambiguous without also printing the repository in which the update happens.

Considering that both multi-remote and submodule fetches are user-facing features, using them in conjunction with `--porcelain` that is intended for scripting purposes is likely not going to be useful in the majority of cases. With that in mind these restrictions feel acceptable. If usecases for either of these come up in the future though it is easy enough to add a new "porcelain-v2" format that adds this information.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/fetch-options.txt |   7 ++
 Documentation/git-fetch.txt     |   9 ++
 builtin/fetch.c                 |  88 ++++++++++++++----
 t/t5574-fetch-output.sh         | 159 ++++++++++++++++++++++++++++++++
 4 files changed, 244 insertions(+), 19 deletions(-)
Show changes to 4 files +244 −19

Documentation/fetch-options.txt, Documentation/git-fetch.txt, builtin/fetch.c, t/t5574-fetch-output.sh

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 622bd84768..41fc7ca3c6 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -78,6 +78,13 @@ linkgit:git-config[1].
 --dry-run::
 	Show what would be done, without making any changes.
 
+--porcelain::
+	Print the output to standard output in an easy-to-parse format for
+	scripts. See section OUTPUT in linkgit:git-fetch[1] for details.
++
+This is incompatible with `--recurse-submodules=[yes|on-demand]` and takes
+precedence over the `fetch.output` config option.
+
 ifndef::git-pull[]
 --[no-]write-fetch-head::
 	Write the list of remote refs fetched in the `FETCH_HEAD`
diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt
index fba66f1460..f123139c58 100644
--- a/Documentation/git-fetch.txt
+++ b/Documentation/git-fetch.txt
@@ -204,6 +204,15 @@ representing the status of a single ref. Each line is of the form:
  <flag> <summary> <from> -> <to> [<reason>]
 -------------------------------
 
+When using `--porcelain`, the output format is intended to be
+machine-parseable. In contrast to the human-readable output formats it
+thus prints to standard output instead of standard error. Each line is
+of the form:
+
+-------------------------------
+<flag> <old-object-id> <new-object-id> <local-reference>
+-------------------------------
+
 The status of up-to-date refs is shown only if the --verbose option is
 used.
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 7a3d620c4c..462fc86b99 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -52,6 +52,7 @@ enum display_format {
 	DISPLAY_FORMAT_UNKNOWN = 0,
 	DISPLAY_FORMAT_FULL,
 	DISPLAY_FORMAT_COMPACT,
+	DISPLAY_FORMAT_PORCELAIN,
 };
 
 struct display_state {
@@ -754,6 +755,9 @@ static void display_state_init(struct display_state *display_state, struct ref *
 		display_state->refcol_width = refcol_width(ref_map,
 							   display_state->format == DISPLAY_FORMAT_COMPACT);
 		break;
+	case DISPLAY_FORMAT_PORCELAIN:
+		/* We don't need to precompute anything here. */
+		break;
 	default:
 		BUG("unexpected display format %d", display_state->format);
 	}
@@ -824,8 +828,12 @@ static void print_compact(struct display_state *display_state,
 static void display_ref_update(struct display_state *display_state, char code,
 			       const char *summary, const char *error,
 			       const char *remote, const char *local,
+			       const struct object_id *old_oid,
+			       const struct object_id *new_oid,
 			       int summary_width)
 {
+	FILE *f = stderr;
+
 	if (verbosity < 0)
 		return;
 
@@ -858,12 +866,17 @@ static void display_ref_update(struct display_state *display_state, char code,
 
 		break;
 	}
+	case DISPLAY_FORMAT_PORCELAIN:
+		strbuf_addf(&display_state->buf, "%c %s %s %s", code,
+			    oid_to_hex(old_oid), oid_to_hex(new_oid), local);
+		f = stdout;
+		break;
 	default:
 		BUG("unexpected display format %d", display_state->format);
 	};
 	strbuf_addch(&display_state->buf, '\n');
 
-	fputs(display_state->buf.buf, stderr);
+	fputs(display_state->buf.buf, f);
 }
 
 static int update_local_ref(struct ref *ref,
@@ -881,7 +894,8 @@ static int update_local_ref(struct ref *ref,
 	if (oideq(&ref->old_oid, &ref->new_oid)) {
 		if (verbosity > 0)
 			display_ref_update(display_state, '=', _("[up to date]"), NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 		return 0;
 	}
 
@@ -894,7 +908,8 @@ static int update_local_ref(struct ref *ref,
 		 */
 		display_ref_update(display_state, '!', _("[rejected]"),
 				   _("can't fetch into checked-out branch"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 
@@ -905,12 +920,14 @@ static int update_local_ref(struct ref *ref,
 			r = s_update_ref("updating tag", ref, transaction, 0);
 			display_ref_update(display_state, r ? '!' : 't', _("[tag update]"),
 					   r ? _("unable to update local ref") : NULL,
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return r;
 		} else {
 			display_ref_update(display_state, '!', _("[rejected]"),
 					   _("would clobber existing tag"),
-					   remote_ref->name, ref->name, summary_width);
+					   remote_ref->name, ref->name,
+					   &ref->old_oid, &ref->new_oid, summary_width);
 			return 1;
 		}
 	}
@@ -943,7 +960,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref(msg, ref, transaction, 0);
 		display_ref_update(display_state, r ? '!' : '*', what,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return r;
 	}
 
@@ -965,7 +983,8 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("fast-forward", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : ' ', quickref.buf,
 				   r ? _("unable to update local ref") : NULL,
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else if (force || ref->force) {
@@ -977,12 +996,14 @@ static int update_local_ref(struct ref *ref,
 		r = s_update_ref("forced-update", ref, transaction, 1);
 		display_ref_update(display_state, r ? '!' : '+', quickref.buf,
 				   r ? _("unable to update local ref") : _("forced update"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		strbuf_release(&quickref);
 		return r;
 	} else {
 		display_ref_update(display_state, '!', _("[rejected]"), _("non-fast-forward"),
-				   remote_ref->name, ref->name, summary_width);
+				   remote_ref->name, ref->name,
+				   &ref->old_oid, &ref->new_oid, summary_width);
 		return 1;
 	}
 }
@@ -1223,7 +1244,9 @@ static int store_updated_refs(struct display_state *display_state,
 				display_ref_update(display_state, '*',
 						   *kind ? kind : "branch", NULL,
 						   rm->name,
-						   "FETCH_HEAD", summary_width);
+						   "FETCH_HEAD",
+						   &rm->new_oid, &rm->old_oid,
+						   summary_width);
 			}
 		}
 	}
@@ -1363,6 +1386,7 @@ static int prune_refs(struct display_state *display_state,
 		for (ref = stale_refs; ref; ref = ref->next) {
 			display_ref_update(display_state, '-', _("[deleted]"), NULL,
 					   _("(none)"), ref->name,
+					   &ref->new_oid, &ref->old_oid,
 					   summary_width);
 			warn_dangling_symref(stderr, dangling_msg, ref->name);
 		}
@@ -1795,7 +1819,8 @@ static int add_remote_or_group(const char *name, struct string_list *list)
 	return 1;
 }
 
-static void add_options_to_argv(struct strvec *argv)
+static void add_options_to_argv(struct strvec *argv,
+				enum display_format format)
 {
 	if (dry_run)
 		strvec_push(argv, "--dry-run");
@@ -1831,6 +1856,8 @@ static void add_options_to_argv(struct strvec *argv)
 		strvec_push(argv, "--ipv6");
 	if (!write_fetch_head)
 		strvec_push(argv, "--no-write-fetch-head");
+	if (format == DISPLAY_FORMAT_PORCELAIN)
+		strvec_pushf(argv, "--porcelain");
 }
 
 /* Fetch multiple remotes in parallel */
@@ -1839,6 +1866,7 @@ struct parallel_fetch_state {
 	const char **argv;
 	struct string_list *remotes;
 	int next, result;
+	enum display_format format;
 };
 
 static int fetch_next_remote(struct child_process *cp,
@@ -1858,7 +1886,7 @@ static int fetch_next_remote(struct child_process *cp,
 	strvec_push(&cp->args, remote);
 	cp->git_cmd = 1;
 
-	if (verbosity >= 0)
+	if (verbosity >= 0 && state->format != DISPLAY_FORMAT_PORCELAIN)
 		printf(_("Fetching %s\n"), remote);
 
 	return 1;
@@ -1890,7 +1918,8 @@ static int fetch_finished(int result, struct strbuf *out,
 	return 0;
 }
 
-static int fetch_multiple(struct string_list *list, int max_children)
+static int fetch_multiple(struct string_list *list, int max_children,
+			  enum display_format format)
 {
 	int i, result = 0;
 	struct strvec argv = STRVEC_INIT;
@@ -1903,10 +1932,10 @@ static int fetch_multiple(struct string_list *list, int max_children)
 
 	strvec_pushl(&argv, "fetch", "--append", "--no-auto-gc",
 		     "--no-write-commit-graph", NULL);
-	add_options_to_argv(&argv);
+	add_options_to_argv(&argv, format);
 
 	if (max_children != 1 && list->nr != 1) {
-		struct parallel_fetch_state state = { argv.v, list, 0, 0 };
+		struct parallel_fetch_state state = { argv.v, list, 0, 0, format };
 		const struct run_process_parallel_opts opts = {
 			.tr2_category = "fetch",
 			.tr2_label = "parallel/fetch",
@@ -1930,7 +1959,7 @@ static int fetch_multiple(struct string_list *list, int max_children)
 
 			strvec_pushv(&cmd.args, argv.v);
 			strvec_push(&cmd.args, name);
-			if (verbosity >= 0)
+			if (verbosity >= 0 && format != DISPLAY_FORMAT_PORCELAIN)
 				printf(_("Fetching %s\n"), name);
 			cmd.git_cmd = 1;
 			if (run_command(&cmd)) {
@@ -2081,6 +2110,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	int fetch_write_commit_graph = -1;
 	int stdin_refspecs = 0;
 	int negotiate_only = 0;
+	int porcelain = 0;
 	int i;
 
 	struct option builtin_fetch_options[] = {
@@ -2115,6 +2145,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
 		OPT_BOOL(0, "dry-run", &dry_run,
 			 N_("dry run")),
+		OPT_BOOL(0, "porcelain", &porcelain, N_("machine-readable output")),
 		OPT_BOOL(0, "write-fetch-head", &write_fetch_head,
 			 N_("write fetched references to the FETCH_HEAD file")),
 		OPT_BOOL('k', "keep", &keep, N_("keep downloaded pack")),
@@ -2220,6 +2251,26 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		fetch_config_from_gitmodules(sfjc, rs);
 	}
 
+
+	if (porcelain) {
+		switch (recurse_submodules_cli) {
+		case RECURSE_SUBMODULES_OFF:
+		case RECURSE_SUBMODULES_DEFAULT:
+			/*
+			 * Reference updates in submodules would be ambiguous
+			 * in porcelain mode, so we reject this combination.
+			 */
+			recurse_submodules = RECURSE_SUBMODULES_OFF;
+			break;
+
+		default:
+			die(_("options '%s' and '%s' cannot be used together"),
+			    "--porcelain", "--recurse-submodules");
+		}
+
+		config.display_format = DISPLAY_FORMAT_PORCELAIN;
+	}
+
 	if (negotiate_only && !negotiation_tip.nr)
 		die(_("--negotiate-only needs one or more --negotiation-tip=*"));
 
@@ -2339,10 +2390,9 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 			max_children = fetch_parallel_config;
 
 		/* TODO should this also die if we have a previous partial-clone? */
-		result = fetch_multiple(&list, max_children);
+		result = fetch_multiple(&list, max_children, config.display_format);
 	}
 
-
 	/*
 	 * This is only needed after fetch_one(), which does not fetch
 	 * submodules by itself.
@@ -2361,7 +2411,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 		if (max_children < 0)
 			max_children = fetch_parallel_config;
 
-		add_options_to_argv(&options);
+		add_options_to_argv(&options, config.display_format);
 		result = fetch_submodules(the_repository,
 					  &options,
 					  submodule_prefix,
diff --git a/t/t5574-fetch-output.sh b/t/t5574-fetch-output.sh
index 9890f6f381..90e6dcb9a7 100755
--- a/t/t5574-fetch-output.sh
+++ b/t/t5574-fetch-output.sh
@@ -61,6 +61,141 @@ test_expect_success 'fetch compact output' '
 	test_cmp expect actual
 '
 
+test_expect_success 'fetch porcelain output' '
+	test_when_finished "rm -rf porcelain" &&
+
+	# Set up a bunch of references that we can use to demonstrate different
+	# kinds of flag symbols in the output format.
+	MAIN_OLD=$(git rev-parse HEAD) &&
+	git branch "fast-forward" &&
+	git branch "deleted-branch" &&
+	git checkout -b force-updated &&
+	test_commit --no-tag force-update-old &&
+	FORCE_UPDATED_OLD=$(git rev-parse HEAD) &&
+	git checkout main &&
+
+	# Clone and pre-seed the repositories. We fetch references into two
+	# namespaces so that we can test that rejected and force-updated
+	# references are reported properly.
+	refspecs="refs/heads/*:refs/unforced/* +refs/heads/*:refs/forced/*" &&
+	git clone . porcelain &&
+	git -C porcelain fetch origin $refspecs &&
+
+	# Now that we have set up the client repositories we can change our
+	# local references.
+	git branch new-branch &&
+	git branch -d deleted-branch &&
+	git checkout fast-forward &&
+	test_commit --no-tag fast-forward-new &&
+	FAST_FORWARD_NEW=$(git rev-parse HEAD) &&
+	git checkout force-updated &&
+	git reset --hard HEAD~ &&
+	test_commit --no-tag force-update-new &&
+	FORCE_UPDATED_NEW=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	- $MAIN_OLD $ZERO_OID refs/forced/deleted-branch
+	- $MAIN_OLD $ZERO_OID refs/unforced/deleted-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/unforced/fast-forward
+	! $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/unforced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/unforced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/forced/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/forced/force-updated
+	* $ZERO_OID $MAIN_OLD refs/forced/new-branch
+	  $MAIN_OLD $FAST_FORWARD_NEW refs/remotes/origin/fast-forward
+	+ $FORCE_UPDATED_OLD $FORCE_UPDATED_NEW refs/remotes/origin/force-updated
+	* $ZERO_OID $MAIN_OLD refs/remotes/origin/new-branch
+	EOF
+
+	# Execute a dry-run fetch first. We do this to assert that the dry-run
+	# and non-dry-run fetches produces the same output. Execution of the
+	# fetch is expected to fail as we have a rejected reference update.
+	test_must_fail git -C porcelain fetch \
+		--porcelain --dry-run --prune origin $refspecs >actual &&
+	test_cmp expect actual &&
+
+	# And now we perform a non-dry-run fetch.
+	test_must_fail git -C porcelain fetch \
+		--porcelain --prune origin $refspecs >actual 2>stderr &&
+	test_cmp expect actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'fetch porcelain with multiple remotes' '
+	test_when_finished "rm -rf porcelain" &&
+
+	git switch --create multiple-remotes &&
+	git clone . porcelain &&
+	git -C porcelain remote add second-remote "$PWD" &&
+	git -C porcelain fetch second-remote &&
+
+	test_commit --no-tag multi-commit &&
+	old_commit=$(git rev-parse HEAD~) &&
+	new_commit=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	  $old_commit $new_commit refs/remotes/origin/multiple-remotes
+	  $old_commit $new_commit refs/remotes/second-remote/multiple-remotes
+	EOF
+
+	git -C porcelain fetch --porcelain --all >actual 2>stderr &&
+	test_cmp expect actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'fetch porcelain refuses to work with submodules' '
+	test_when_finished "rm -rf porcelain" &&
+
+	cat >expect <<-EOF &&
+	fatal: options ${SQ}--porcelain${SQ} and ${SQ}--recurse-submodules${SQ} cannot be used together
+	EOF
+
+	git init porcelain &&
+	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=yes 2>stderr &&
+	test_cmp expect stderr &&
+
+	test_must_fail git -C porcelain fetch --porcelain --recurse-submodules=on-demand 2>stderr &&
+	test_cmp expect stderr
+'
+
+test_expect_success 'fetch porcelain overrides fetch.output config' '
+	test_when_finished "rm -rf porcelain" &&
+
+	git switch --create config-override &&
+	git clone . porcelain &&
+	test_commit new-commit &&
+	old_commit=$(git rev-parse HEAD~) &&
+	new_commit=$(git rev-parse HEAD) &&
+
+	cat >expect <<-EOF &&
+	  $old_commit $new_commit refs/remotes/origin/config-override
+	* $ZERO_OID $new_commit refs/tags/new-commit
+	EOF
+
+	git -C porcelain -c fetch.output=compact fetch --porcelain >stdout 2>stderr &&
+	test_must_be_empty stderr &&
+	test_cmp expect stdout
+'
+
+test_expect_success 'fetch --no-porcelain overrides previous --porcelain' '
+	test_when_finished "rm -rf no-porcelain" &&
+
+	git switch --create no-porcelain &&
+	git clone . no-porcelain &&
+	test_commit --no-tag no-porcelain &&
+	old_commit=$(git rev-parse --short HEAD~) &&
+	new_commit=$(git rev-parse --short HEAD) &&
+
+	cat >expect <<-EOF &&
+	From $(test-tool path-utils real_path .)/.
+	   $old_commit..$new_commit  no-porcelain -> origin/no-porcelain
+	EOF
+
+	git -C no-porcelain fetch --porcelain --no-porcelain >stdout 2>stderr &&
+	test_cmp expect stderr &&
+	test_must_be_empty stdout
+'
+
 test_expect_success 'fetch output with HEAD' '
 	test_when_finished "rm -rf head" &&
 	git clone . head &&
@@ -90,6 +225,30 @@ test_expect_success 'fetch output with HEAD' '
 	test_cmp expect actual.err
 '
 
+test_expect_success 'fetch porcelain output with HEAD' '
+	test_when_finished "rm -rf head" &&
+	git clone . head &&
+	COMMIT_ID=$(git rev-parse HEAD) &&
+
+	git -C head fetch --porcelain --dry-run origin HEAD >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID FETCH_HEAD
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --porcelain origin HEAD >actual &&
+	test_cmp expect actual &&
+
+	git -C head fetch --porcelain --dry-run origin HEAD:foo >actual &&
+	cat >expect <<-EOF &&
+	* $ZERO_OID $COMMIT_ID refs/heads/foo
+	EOF
+	test_cmp expect actual &&
+
+	git -C head fetch --porcelain origin HEAD:foo >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'fetch output with object ID' '
 	test_when_finished "rm -rf object-id" &&
 	git clone . object-id &&
-- 
2.40.1
Glen Choo· May 12, 2023, 01:02 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v5 9/9] fetch: introduce machine-parseable "porcelain" output format

Patrick Steinhardt <ps@pks.im> writes:
Show 18 quoted lines
> +test_expect_success 'fetch --no-porcelain overrides previous --porcelain' '
> +	test_when_finished "rm -rf no-porcelain" &&
> +
> +	git switch --create no-porcelain &&
> +	git clone . no-porcelain &&
> +	test_commit --no-tag no-porcelain &&
> +	old_commit=$(git rev-parse --short HEAD~) &&
> +	new_commit=$(git rev-parse --short HEAD) &&
> +
> +	cat >expect <<-EOF &&
> +	From $(test-tool path-utils real_path .)/.
> +	   $old_commit..$new_commit  no-porcelain -> origin/no-porcelain
> +	EOF
> +
> +	git -C no-porcelain fetch --porcelain --no-porcelain >stdout 2>stderr &&
> +	test_cmp expect stderr &&
> +	test_must_be_empty stdout
> +'

(This shouldn't block the series) As a matter of taste, it feels like overkill to construct the whole output to check if --no-porcelain wins. Maybe in a future cleanup, we would go through the tests in this file and see where it is really important that we have the exact output and where it is not.

Junio C Hamano· May 10, 2023, 18:05 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v5 0/9] fetch: introduce machine-parseable output

Patrick Steinhardt <ps@pks.im> writes:
>     - Patch 3/9: Added a test to verify that `git fetch -c fetch.output`
>       without a value set fails as expected. Also dropped the tests that
>       checked whether stdout was empty.

Thank you for the attention to this small detail of valueless configuration variable definition.

>     - Patch 4/9: Reformulated the commit message to treat the missing
>       left-hand side of displayed references as an inconsistency instead
>       of a bug. I've also added a testcase to verify that direct OID
>       fetches continue to work as expected.

Again, the direct OID fetch is a good thing to test here. I noticed that the test added here insists that the standard output stream is empty when the command errors out, which is not consistent with [3/9] above.

>     - Patch 5/9: New patch that makes calculation of the table width for
>       displayed reference updates self-contained in `refcol_width()`.
>       This is a preparatory refactoring that should make patch 6/9
>       easier to review.

It is an excellent idea to avoid calls to refcol_width() for each ref that gets shown and make the helper responsible for computing the required maxwidth. The result indeed has become easier to follow as you mentioned in your response to my review on the previous round.

Show 5 quoted lines
>     - Patch 7/9: Refactored the code to parse the "fetch.output" config
>       variable inside of `git_fetch_config()` before we parse command
>       line options. Also fixed that the commit message was still
>       referring to `--output-format=porcelain` instead of the new
>       `--porcelain` switch.

As a standalone step, it leaves an impression that the step makes the way we handle the output-format configuration variable inconsistent with the way we handle the other configuration variables, but I think it is a good place to stop for the purpose of this topic. It lays a good foundation for future clean-up after the dust settles from this topic---we may want to move global variables assigned in the git_fetch_config() into the fetch_config structure.

>     - Patch 9/9: The `--porcelain` option is now a simple `OPT_BOOL()`
>       that can be negated. Added a test that `--no-porcelain` works as
>       expected.

OK, this time the familiar "prepare a variable to its default, let config callback to overwrite it by reading configuration variables, and then let the command line option override it" is used and the result easy to follow. I do not think .display_format is never assigned DISPLAY_FORMAT_UNKNOWN with this change, so [6/9] could lose the value from the enum, I think. The defensive switch statement that has BUG() to notice an erroneous caller that pass values other than DISPLAY_FORMAT_{FULL,COMPACT} is still a good idea.

Looking good.
Thanks.
Patrick Steinhardt· May 11, 2023, 11:05 UTC · re: Junio C Hamano · lore

Re: [PATCH v5 0/9] fetch: introduce machine-parseable output

On Wed, May 10, 2023 at 11:05:19AM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
[snip]
Show 9 quoted lines
> >     - Patch 4/9: Reformulated the commit message to treat the missing
> >       left-hand side of displayed references as an inconsistency instead
> >       of a bug. I've also added a testcase to verify that direct OID
> >       fetches continue to work as expected.
> 
> Again, the direct OID fetch is a good thing to test here.  I noticed
> that the test added here insists that the standard output stream is
> empty when the command errors out, which is not consistent with [3/9]
> above.

I think you misread the test: yes, we do test stdout and stderr separately and in many cases assert that stdout is in fact empty. But none of the added tests are about failing commands. So given that:

    - The added tests explicitly are about verifying the output format.
    - The distinction between stdout and stderr matters.
    - The distinction matters even more with the addition of
      `--porcelain`.

I think that explicitly verifing both output streams is the correct thing to do.

Show 13 quoted lines
> >     - Patch 9/9: The `--porcelain` option is now a simple `OPT_BOOL()`
> >       that can be negated. Added a test that `--no-porcelain` works as
> >       expected.
> 
> OK, this time the familiar "prepare a variable to its default, let
> config callback to overwrite it by reading configuration variables,
> and then let the command line option override it" is used and the
> result easy to follow.  I do not think .display_format is never
> assigned DISPLAY_FORMAT_UNKNOWN with this change, so [6/9] could
> lose the value from the enum, I think.  The defensive switch
> statement that has BUG() to notice an erroneous caller that pass
> values other than DISPLAY_FORMAT_{FULL,COMPACT} is still a good
> idea.

Ah, right, `DISPLAY_FORMAT_UNKNOWN` isn't really needed anymore. I think it's still good to have valid values of the enum start with `1` so that it becomes easier to detect cases where we accidentally use a default initialized variable. But that can be achieved without giving the default value an explicit name.

I'll refrain from sending a new version just to remove this constant as it doesn't really feel worth it, though. But please, let me know in case you disagree and I'll send an updated version.

Thanks Patrick

Junio C Hamano· May 11, 2023, 16:53 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v5 0/9] fetch: introduce machine-parseable output

Patrick Steinhardt <ps@pks.im> writes:
Show 16 quoted lines
> On Wed, May 10, 2023 at 11:05:19AM -0700, Junio C Hamano wrote:
>> Patrick Steinhardt <ps@pks.im> writes:
> [snip]
>> >     - Patch 4/9: Reformulated the commit message to treat the missing
>> >       left-hand side of displayed references as an inconsistency instead
>> >       of a bug. I've also added a testcase to verify that direct OID
>> >       fetches continue to work as expected.
>> 
>> Again, the direct OID fetch is a good thing to test here.  I noticed
>> that the test added here insists that the standard output stream is
>> empty when the command errors out, which is not consistent with [3/9]
>> above.
>
> I think you misread the test: yes, we do test stdout and stderr
> separately and in many cases assert that stdout is in fact empty. But
> none of the added tests are about failing commands.

Ah, you are absolutely right. Sorry for the noise---I do not have an objection to ensure that program output in the successful case is predictable. My main concern is to avoid giving unneeded assurance in the failing case.

Show 11 quoted lines
>> assigned DISPLAY_FORMAT_UNKNOWN with this change, so [6/9] could
>> lose the value from the enum, I think.  The defensive switch
>> statement that has BUG() to notice an erroneous caller that pass
>> values other than DISPLAY_FORMAT_{FULL,COMPACT} is still a good
>> idea.
>
> Ah, right, `DISPLAY_FORMAT_UNKNOWN` isn't really needed anymore. I think
> it's still good to have valid values of the enum start with `1` so that
> it becomes easier to detect cases where we accidentally use a default
> initialized variable. But that can be achieved without giving the
> default value an explicit name.

Unless we explicitly take advantage of UNKNOWN being 0 and perform clever defaulting, the result is far safer if you get rid of it, and instead give 0 to a real choice that is used as the default, for two reasons: (1) naturally 0 initialization work fine, (2) we have one less enum constant people (or editor's auto-completer) assign to a variable by mistake instead of a real one they intended to. I agree with the last sentence of your paragraph above.

Felipe Contreras· May 11, 2023, 17:24 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v5 0/9] fetch: introduce machine-parseable output

Patrick Steinhardt wrote:
> On Wed, May 10, 2023 at 11:05:19AM -0700, Junio C Hamano wrote:
Show 15 quoted lines
> > OK, this time the familiar "prepare a variable to its default, let
> > config callback to overwrite it by reading configuration variables,
> > and then let the command line option override it" is used and the
> > result easy to follow.  I do not think .display_format is never
> > assigned DISPLAY_FORMAT_UNKNOWN with this change, so [6/9] could
> > lose the value from the enum, I think.  The defensive switch
> > statement that has BUG() to notice an erroneous caller that pass
> > values other than DISPLAY_FORMAT_{FULL,COMPACT} is still a good
> > idea.
> 
> Ah, right, `DISPLAY_FORMAT_UNKNOWN` isn't really needed anymore. I think
> it's still good to have valid values of the enum start with `1` so that
> it becomes easier to detect cases where we accidentally use a default
> initialized variable. But that can be achieved without giving the
> default value an explicit name.

I think it's standard to have an element like that. It could be UNKNOWN, NULL, INVALID, or even DEFAULT.

For example, you could have code like this:
 if (format == DISPLAY_FORMAT_DEFAULT)
        format = DISPLAY_FORMAT_FULL;

This would distinguish cases in which the user did not specify a display format and we choose a default, from those where the user explicitely chose the format that happens to be the default.

-- 
Felipe Contreras
Glen Choo· May 12, 2023, 01:09 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v5 0/9] fetch: introduce machine-parseable output

Patrick Steinhardt <ps@pks.im> writes:
Show 27 quoted lines
> Changes compared to v4:
>
>     - Patch 1/9: Simplified the test as proposed by Junio and Glen.
>
>     - Patch 3/9: Added a test to verify that `git fetch -c fetch.output`
>       without a value set fails as expected. Also dropped the tests that
>       checked whether stdout was empty.
>
>     - Patch 4/9: Reformulated the commit message to treat the missing
>       left-hand side of displayed references as an inconsistency instead
>       of a bug. I've also added a testcase to verify that direct OID
>       fetches continue to work as expected.
>
>     - Patch 5/9: New patch that makes calculation of the table width for
>       displayed reference updates self-contained in `refcol_width()`.
>       This is a preparatory refactoring that should make patch 6/9
>       easier to review.
>
>     - Patch 7/9: Refactored the code to parse the "fetch.output" config
>       variable inside of `git_fetch_config()` before we parse command
>       line options. Also fixed that the commit message was still
>       referring to `--output-format=porcelain` instead of the new
>       `--porcelain` switch.
>
>     - Patch 9/9: The `--porcelain` option is now a simple `OPT_BOOL()`
>       that can be negated. Added a test that `--no-porcelain` works as
>       expected.

I didn't spot any blocking issues in this version, and the various improvements (especially 6-7/9) are really welcome. I also read through Junio's comments, but I didn't spot anything that I thought should block the series, so I'm happy to leave

  Reviewed-by: Glen Choo <chooglen@google.com>
Patrick Steinhardt· May 12, 2023, 07:16 UTC · re: Glen Choo · lore

Re: [PATCH v5 0/9] fetch: introduce machine-parseable output

On Thu, May 11, 2023 at 06:09:03PM -0700, Glen Choo wrote:
Show 36 quoted lines
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Changes compared to v4:
> >
> >     - Patch 1/9: Simplified the test as proposed by Junio and Glen.
> >
> >     - Patch 3/9: Added a test to verify that `git fetch -c fetch.output`
> >       without a value set fails as expected. Also dropped the tests that
> >       checked whether stdout was empty.
> >
> >     - Patch 4/9: Reformulated the commit message to treat the missing
> >       left-hand side of displayed references as an inconsistency instead
> >       of a bug. I've also added a testcase to verify that direct OID
> >       fetches continue to work as expected.
> >
> >     - Patch 5/9: New patch that makes calculation of the table width for
> >       displayed reference updates self-contained in `refcol_width()`.
> >       This is a preparatory refactoring that should make patch 6/9
> >       easier to review.
> >
> >     - Patch 7/9: Refactored the code to parse the "fetch.output" config
> >       variable inside of `git_fetch_config()` before we parse command
> >       line options. Also fixed that the commit message was still
> >       referring to `--output-format=porcelain` instead of the new
> >       `--porcelain` switch.
> >
> >     - Patch 9/9: The `--porcelain` option is now a simple `OPT_BOOL()`
> >       that can be negated. Added a test that `--no-porcelain` works as
> >       expected.
> 
> I didn't spot any blocking issues in this version, and the various
> improvements (especially 6-7/9) are really welcome. I also read through
> Junio's comments, but I didn't spot anything that I thought should block
> the series, so I'm happy to leave
> 
>   Reviewed-by: Glen Choo <chooglen@google.com>
Thanks a lot for your reviews!
Patrick

← back to recent threads