threads / patch / 61536

patchpromisor-remote: add promisor.quiet configuration option

Subject: [PATCH] promisor-remote: add promisor.quiet configuration option

## tl;dr

10 messages between May 23, 2024 and May 29, 2024. Diffs are folded; open one to read it.

replies: 9people: 3as markdown or json

Tom Hughes· May 23, 2024, 13:19 UTC · lore

Add a configuration optione to allow output from the promisor fetching objects to be suppressed/

This allows us to stop commands like git blame being swamped with progress messages and gc notifications from the promisor when used in a partial clone.

Signed-off-by: Tom Hughes <tom@compton.nu>
---
 Documentation/config.txt          |  2 ++
 Documentation/config/promisor.txt |  3 ++
 promisor-remote.c                 |  3 ++
 t/t0410-partial-clone.sh          | 47 +++++++++++++++++++++++++++++++
 4 files changed, 55 insertions(+)
 create mode 100644 Documentation/config/promisor.txt
Show changes to 4 files +55 −0

Documentation/config.txt, Documentation/config/promisor.txt, promisor-remote.c, t/t0410-partial-clone.sh

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 70b448b132..6cae835db9 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -487,6 +487,8 @@ include::config/pager.txt[]
 
 include::config/pretty.txt[]
 
+include::config/promisor.txt[]
+
 include::config/protocol.txt[]
 
 include::config/pull.txt[]
diff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt
new file mode 100644
index 0000000000..98c5cb2ec2
--- /dev/null
+++ b/Documentation/config/promisor.txt
@@ -0,0 +1,3 @@
+promisor.quiet::
+	If set to "true" assume `--quiet` when fetching additional
+	objects for a partial clone.
diff --git a/promisor-remote.c b/promisor-remote.c
index b414922c44..2ca7c2ae48 100644
--- a/promisor-remote.c
+++ b/promisor-remote.c
@@ -23,6 +23,7 @@ static int fetch_objects(struct repository *repo,
 	struct child_process child = CHILD_PROCESS_INIT;
 	int i;
 	FILE *child_in;
+	int quiet;
 
 	if (git_env_bool(NO_LAZY_FETCH_ENVIRONMENT, 0)) {
 		static int warning_shown;
@@ -41,6 +42,8 @@ static int fetch_objects(struct repository *repo,
 		     "fetch", remote_name, "--no-tags",
 		     "--no-write-fetch-head", "--recurse-submodules=no",
 		     "--filter=blob:none", "--stdin", NULL);
+	if (!git_config_get_bool("promisor.quiet", &quiet) && quiet)
+		strvec_push(&child.args, "--quiet");
 	if (start_command(&child))
 		die(_("promisor-remote: unable to fork off fetch subprocess"));
 	child_in = xfdopen(child.in, "w");
diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh
index 88a66f0904..99257c3792 100755
--- a/t/t0410-partial-clone.sh
+++ b/t/t0410-partial-clone.sh
@@ -3,6 +3,7 @@
 test_description='partial clone'
 
 . ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-terminal.sh
 
 # missing promisor objects cause repacks which write bitmaps to fail
 GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
@@ -689,6 +690,52 @@ test_expect_success 'lazy-fetch when accessing object not in the_repository' '
 	! grep "[?]$FILE_HASH" out
 '
 
+test_expect_success TTY 'promisor.quiet=false works' '
+	rm -rf server server2 repo &&
+	rm -rf server server3 repo &&
+	test_create_repo server &&
+	test_commit -C server foo &&
+	git -C server repack -a -d --write-bitmap-index &&
+
+	git clone "file://$(pwd)/server" repo &&
+	git hash-object repo/foo.t >blobhash &&
+	rm -rf repo/.git/objects/* &&
+
+	git -C server config uploadpack.allowanysha1inwant 1 &&
+	git -C server config uploadpack.allowfilter 1 &&
+	git -C repo config core.repositoryformatversion 1 &&
+	git -C repo config extensions.partialclone "origin" &&
+	git -C repo config promisor.quiet "false" &&
+
+	test_terminal git -C repo cat-file -p $(cat blobhash) 2>err &&
+
+	# Ensure that progress messages are written
+	grep "Receiving objects" err
+'
+
+test_expect_success TTY 'promisor.quiet=true works' '
+	rm -rf server server2 repo &&
+	rm -rf server server3 repo &&
+	test_create_repo server &&
+	test_commit -C server foo &&
+	git -C server repack -a -d --write-bitmap-index &&
+
+	git clone "file://$(pwd)/server" repo &&
+	git hash-object repo/foo.t >blobhash &&
+	rm -rf repo/.git/objects/* &&
+
+	git -C server config uploadpack.allowanysha1inwant 1 &&
+	git -C server config uploadpack.allowfilter 1 &&
+	git -C repo config core.repositoryformatversion 1 &&
+	git -C repo config extensions.partialclone "origin" &&
+	git -C repo config promisor.quiet "true" &&
+
+	test_terminal git -C repo cat-file -p $(cat blobhash) 2>err &&
+
+	# Ensure that no progress messages are written
+	! grep "Receiving objects" err
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 

-- 
2.45.1
Junio C Hamano· May 23, 2024, 22:23 UTC · re: Tom Hughes · lore

Re: [PATCH] promisor-remote: add promisor.quiet configuration option

Tom Hughes <tom@compton.nu> writes:
> Add a configuration optione to allow output from the promisor
> fetching objects to be suppressed/
"optione" -> "option", "suppressed/" -> "suppressed.".
> This allows us to stop commands like git blame being swamped
> with progress messages and gc notifications from the promisor
> when used in a partial clone.
"git blame" -> "'git blame'", perhaps.

It is an interesting observation. I thought "git blame" was quite bad at streaming (i.e., until it learned the origin of each and every line, it never produced any output the user asked for), which actually would make it a non issue that the output the user wanted gets mixed with the progress messages and other garbage. Unless the user understands that "git blame" is not spending time itself, but is waiting for necessary blobs to be fetched from the promisor, and is expected to wait unusally longer than the fully local case, having to stare at a blank/unchanging screen would make it uneasy for the end-user and that is why we have progress eye-candy.

I am OK for promisor.quiet being optional, but I am torn when I imagine what comes next. On one hand, I myself probably would find it neat to make these lazy fetches happen completely silently as if nothing strange is happening from the point of view of end-users (except for some operations may be unusually slow compared to fully local repository). On the other hand, I suspect people will be tempted to push it to be on by default at which time it may hurt unsuspecting (new) users who may have been helped by progress bars.

Show 9 quoted lines
> diff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt
> new file mode 100644
> index 0000000000..98c5cb2ec2
> --- /dev/null
> +++ b/Documentation/config/promisor.txt
> @@ -0,0 +1,3 @@
> +promisor.quiet::
> +	If set to "true" assume `--quiet` when fetching additional
> +	objects for a partial clone.
OK.
> diff --git a/promisor-remote.c b/promisor-remote.c
The implementation is absolutely trivial and straight-forward.
> +test_expect_success TTY 'promisor.quiet=false works' '

Do not say "works"---recall the best practice of writing a good bug reports. Stating your expectation more explicitly, e.g. "shows progress messages" or somesuch.

> +	rm -rf server server2 repo &&
> +	rm -rf server server3 repo &&
Why remove the same thing twice?
Show 6 quoted lines
> +	test_create_repo server &&
> +	test_commit -C server foo &&
> +	git -C server repack -a -d --write-bitmap-index &&
> +
> +	git clone "file://$(pwd)/server" repo &&
> +	git hash-object repo/foo.t >blobhash &&
Do you need a temporary file, or would
	blobhash=$(git hash-object repo/foo.t) &&
work just fine?  Of course you'd later have to say
	... git -C repo cat-file -p $blobhash

instead of "$(cat blobhash)". Even simpler, I wonder if you can remove this hash-object invocation, and then do

	... git -C repo cat-file -p :foo.t
> +	rm -rf repo/.git/objects/* &&
This! IS! BAD! for the reason stated later ...
> +	git -C server config uploadpack.allowanysha1inwant 1 &&
> +	git -C server config uploadpack.allowfilter 1 &&
... but these are OK and expected, ...
> +	git -C repo config core.repositoryformatversion 1 &&
> +	git -C repo config extensions.partialclone "origin" &&

... and this is way too different from what would happen in the real life.

I'd prefer not to see manual destruction of $GIT_DIR/objects/* or manual futzing of repository format version and extensions. These configuration variables are *NOT* for end-users to futz with, and the tests should not be doing so either.

Can't we prepare the "repo" only by creating a partial clone in an usual way?

> +	git -C repo config promisor.quiet "false" &&
This of course is good, as this is what the test wants to check.
> +	test_terminal git -C repo cat-file -p $(cat blobhash) 2>err &&

It seems that exactly the same set of comments apply to the next one, so I'll refrain from repeating myself.

Thanks.
Show 25 quoted lines
> +test_expect_success TTY 'promisor.quiet=true works' '
> +	rm -rf server server2 repo &&
> +	rm -rf server server3 repo &&
> +	test_create_repo server &&
> +	test_commit -C server foo &&
> +	git -C server repack -a -d --write-bitmap-index &&
> +
> +	git clone "file://$(pwd)/server" repo &&
> +	git hash-object repo/foo.t >blobhash &&
> +	rm -rf repo/.git/objects/* &&
> +
> +	git -C server config uploadpack.allowanysha1inwant 1 &&
> +	git -C server config uploadpack.allowfilter 1 &&
> +	git -C repo config core.repositoryformatversion 1 &&
> +	git -C repo config extensions.partialclone "origin" &&
> +	git -C repo config promisor.quiet "true" &&
> +
> +	test_terminal git -C repo cat-file -p $(cat blobhash) 2>err &&
> +
> +	# Ensure that no progress messages are written
> +	! grep "Receiving objects" err
> +'
> +
>  . "$TEST_DIRECTORY"/lib-httpd.sh
>  start_httpd
Tom Hughes· May 24, 2024, 08:31 UTC · re: Junio C Hamano · lore

Re: [PATCH] promisor-remote: add promisor.quiet configuration option

On 23/05/2024 23:23, Junio C Hamano wrote:
Show 10 quoted lines
> It is an interesting observation.  I thought "git blame" was quite
> bad at streaming (i.e., until it learned the origin of each and
> every line, it never produced any output the user asked for), which
> actually would make it a non issue that the output the user wanted
> gets mixed with the progress messages and other garbage.  Unless the
> user understands that "git blame" is not spending time itself, but
> is waiting for necessary blobs to be fetched from the promisor, and
> is expected to wait unusally longer than the fully local case,
> having to stare at a blank/unchanging screen would make it uneasy
> for the end-user and that is why we have progress eye-candy.

Blame actually has it's own progress message that counts the number of lines analysed which gets interrupted by the progress messages from the promisor.

Something like "git log -S" behaves a bit differently - it doesn't have progress and because it's using a pager by default that causes the promisor progress to be suppressed because stderr is no longer a terminal but you do still get lots of background gc notifications.

Show 8 quoted lines
> I am OK for promisor.quiet being optional, but I am torn when I
> imagine what comes next.  On one hand, I myself probably would find
> it neat to make these lazy fetches happen completely silently as if
> nothing strange is happening from the point of view of end-users
> (except for some operations may be unusually slow compared to fully
> local repository).  On the other hand, I suspect people will be
> tempted to push it to be on by default at which time it may hurt
> unsuspecting (new) users who may have been helped by progress bars.

I do agree that it's hard to know what the right thing to do is here or even to know the full scope of the effect.

I'll update the patch to address the specific review comments.
Tom
-- 
Tom Hughes (tom@compton.nu)
http://compton.nu/
Tom Hughes· May 24, 2024, 09:09 UTC · re: Tom Hughes · lore

[PATCH v2] promisor-remote: add promisor.quiet configuration option

Add a configuration option to allow output from the promisor fetching objects to be suppressed.

This allows us to stop commands like 'git blame' being swamped with progress messages and gc notifications from the promisor when used in a partial clone.

Signed-off-by: Tom Hughes <tom@compton.nu>
---
 Documentation/config.txt          |  2 ++
 Documentation/config/promisor.txt |  3 +++
 promisor-remote.c                 |  3 +++
 t/t0410-partial-clone.sh          | 37 +++++++++++++++++++++++++++++++
 4 files changed, 45 insertions(+)
 create mode 100644 Documentation/config/promisor.txt
Show changes to 4 files +45 −0

Documentation/config.txt, Documentation/config/promisor.txt, promisor-remote.c, t/t0410-partial-clone.sh

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 70b448b132..6cae835db9 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -487,6 +487,8 @@ include::config/pager.txt[]
 
 include::config/pretty.txt[]
 
+include::config/promisor.txt[]
+
 include::config/protocol.txt[]
 
 include::config/pull.txt[]
diff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt
new file mode 100644
index 0000000000..98c5cb2ec2
--- /dev/null
+++ b/Documentation/config/promisor.txt
@@ -0,0 +1,3 @@
+promisor.quiet::
+	If set to "true" assume `--quiet` when fetching additional
+	objects for a partial clone.
diff --git a/promisor-remote.c b/promisor-remote.c
index b414922c44..2ca7c2ae48 100644
--- a/promisor-remote.c
+++ b/promisor-remote.c
@@ -23,6 +23,7 @@ static int fetch_objects(struct repository *repo,
 	struct child_process child = CHILD_PROCESS_INIT;
 	int i;
 	FILE *child_in;
+	int quiet;
 
 	if (git_env_bool(NO_LAZY_FETCH_ENVIRONMENT, 0)) {
 		static int warning_shown;
@@ -41,6 +42,8 @@ static int fetch_objects(struct repository *repo,
 		     "fetch", remote_name, "--no-tags",
 		     "--no-write-fetch-head", "--recurse-submodules=no",
 		     "--filter=blob:none", "--stdin", NULL);
+	if (!git_config_get_bool("promisor.quiet", &quiet) && quiet)
+		strvec_push(&child.args, "--quiet");
 	if (start_command(&child))
 		die(_("promisor-remote: unable to fork off fetch subprocess"));
 	child_in = xfdopen(child.in, "w");
diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh
index 88a66f0904..2957160efa 100755
--- a/t/t0410-partial-clone.sh
+++ b/t/t0410-partial-clone.sh
@@ -3,6 +3,7 @@
 test_description='partial clone'
 
 . ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-terminal.sh
 
 # missing promisor objects cause repacks which write bitmaps to fail
 GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
@@ -689,6 +690,42 @@ test_expect_success 'lazy-fetch when accessing object not in the_repository' '
 	! grep "[?]$FILE_HASH" out
 '
 
+test_expect_success TTY 'promisor.quiet=false shows progress messages' '
+	rm -rf server repo &&
+	test_create_repo server &&
+	test_commit -C server foo &&
+	git -C server rm foo.t &&
+	git -C server commit -m remove &&
+	git -C server config uploadpack.allowanysha1inwant 1 &&
+	git -C server config uploadpack.allowfilter 1 &&
+
+	git clone --filter=blob:none "file://$(pwd)/server" repo &&
+	git -C repo config promisor.quiet "false" &&
+
+	test_terminal git -C repo cat-file -p foo:foo.t 2>err &&
+
+	# Ensure that progress messages are written
+	grep "Receiving objects" err
+'
+
+test_expect_success TTY 'promisor.quiet=true does not show progress messages' '
+	rm -rf server repo &&
+	test_create_repo server &&
+	test_commit -C server foo &&
+	git -C server rm foo.t &&
+	git -C server commit -m remove &&
+	git -C server config uploadpack.allowanysha1inwant 1 &&
+	git -C server config uploadpack.allowfilter 1 &&
+
+	git clone --filter=blob:none "file://$(pwd)/server" repo &&
+	git -C repo config promisor.quiet "true" &&
+
+	test_terminal git -C repo cat-file -p foo:foo.t 2>err &&
+
+	# Ensure that no progress messages are written
+	! grep "Receiving objects" err
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 

-- 
2.45.1
Junio C Hamano· May 24, 2024, 18:06 UTC · re: Tom Hughes · lore

Re: [PATCH v2] promisor-remote: add promisor.quiet configuration option

Tom Hughes <tom@compton.nu> writes:
Show 35 quoted lines
> +test_expect_success TTY 'promisor.quiet=false shows progress messages' '
> +	rm -rf server repo &&
> +	test_create_repo server &&
> +	test_commit -C server foo &&
> +	git -C server rm foo.t &&
> +	git -C server commit -m remove &&
> +	git -C server config uploadpack.allowanysha1inwant 1 &&
> +	git -C server config uploadpack.allowfilter 1 &&
> +
> +	git clone --filter=blob:none "file://$(pwd)/server" repo &&
> +	git -C repo config promisor.quiet "false" &&
> +
> +	test_terminal git -C repo cat-file -p foo:foo.t 2>err &&
> +
> +	# Ensure that progress messages are written
> +	grep "Receiving objects" err
> +'
> +
> +test_expect_success TTY 'promisor.quiet=true does not show progress messages' '
> +	rm -rf server repo &&
> +	test_create_repo server &&
> +	test_commit -C server foo &&
> +	git -C server rm foo.t &&
> +	git -C server commit -m remove &&
> +	git -C server config uploadpack.allowanysha1inwant 1 &&
> +	git -C server config uploadpack.allowfilter 1 &&
> +
> +	git clone --filter=blob:none "file://$(pwd)/server" repo &&
> +	git -C repo config promisor.quiet "true" &&
> +
> +	test_terminal git -C repo cat-file -p foo:foo.t 2>err &&
> +
> +	# Ensure that no progress messages are written
> +	! grep "Receiving objects" err
> +'
Wouldn't we want to see this as three (or four) tests,
 (1) "setup" that prepares the server end
 (2) "quiet=false" test that
     - makes a partial clone, 
     - sets .quiet to false, 
     - runs cat-file -p,
     - makes sure that the lazy fetching is chatty.
 (3) "quiet=true" test, which is the same as (2) except that it sets
     .quiet to true and expects the lazy fetching to be silent.
 (4) "quiet=unconfigured" test, which is the same as (2) except that
     it leaves .quiet unconfigured.

Other than that, looking very much nicer than the previous round that manually mucked with internal implementation on the client end.

Thanks.
Tom Hughes· May 25, 2024, 10:10 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] promisor-remote: add promisor.quiet configuration option

On 24/05/2024 19:06, Junio C Hamano wrote:
Show 15 quoted lines
> Wouldn't we want to see this as three (or four) tests,
> 
>   (1) "setup" that prepares the server end
> 
>   (2) "quiet=false" test that
>       - makes a partial clone,
>       - sets .quiet to false,
>       - runs cat-file -p,
>       - makes sure that the lazy fetching is chatty.
> 
>   (3) "quiet=true" test, which is the same as (2) except that it sets
>       .quiet to true and expects the lazy fetching to be silent.
> 
>   (4) "quiet=unconfigured" test, which is the same as (2) except that
>       it leaves .quiet unconfigured.
Sounds good to me. I've sent a v3 with those changes.
Tom
-- 
Tom Hughes (tom@compton.nu)
http://compton.nu/
Tom Hughes· May 25, 2024, 10:09 UTC · re: Tom Hughes · lore

[PATCH v3] promisor-remote: add promisor.quiet configuration option

Add a configuration option to allow output from the promisor fetching objects to be suppressed.

This allows us to stop commands like 'git blame' being swamped with progress messages and gc notifications from the promisor when used in a partial clone.

Signed-off-by: Tom Hughes <tom@compton.nu>
---
 Documentation/config.txt          |  2 ++
 Documentation/config/promisor.txt |  3 +++
 promisor-remote.c                 |  3 +++
 t/t0410-partial-clone.sh          | 43 +++++++++++++++++++++++++++++++
 4 files changed, 51 insertions(+)
 create mode 100644 Documentation/config/promisor.txt
Show changes to 4 files +51 −0

Documentation/config.txt, Documentation/config/promisor.txt, promisor-remote.c, t/t0410-partial-clone.sh

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 70b448b132..6cae835db9 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -487,6 +487,8 @@ include::config/pager.txt[]
 
 include::config/pretty.txt[]
 
+include::config/promisor.txt[]
+
 include::config/protocol.txt[]
 
 include::config/pull.txt[]
diff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt
new file mode 100644
index 0000000000..98c5cb2ec2
--- /dev/null
+++ b/Documentation/config/promisor.txt
@@ -0,0 +1,3 @@
+promisor.quiet::
+	If set to "true" assume `--quiet` when fetching additional
+	objects for a partial clone.
diff --git a/promisor-remote.c b/promisor-remote.c
index b414922c44..2ca7c2ae48 100644
--- a/promisor-remote.c
+++ b/promisor-remote.c
@@ -23,6 +23,7 @@ static int fetch_objects(struct repository *repo,
 	struct child_process child = CHILD_PROCESS_INIT;
 	int i;
 	FILE *child_in;
+	int quiet;
 
 	if (git_env_bool(NO_LAZY_FETCH_ENVIRONMENT, 0)) {
 		static int warning_shown;
@@ -41,6 +42,8 @@ static int fetch_objects(struct repository *repo,
 		     "fetch", remote_name, "--no-tags",
 		     "--no-write-fetch-head", "--recurse-submodules=no",
 		     "--filter=blob:none", "--stdin", NULL);
+	if (!git_config_get_bool("promisor.quiet", &quiet) && quiet)
+		strvec_push(&child.args, "--quiet");
 	if (start_command(&child))
 		die(_("promisor-remote: unable to fork off fetch subprocess"));
 	child_in = xfdopen(child.in, "w");
diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh
index 88a66f0904..8d468eb170 100755
--- a/t/t0410-partial-clone.sh
+++ b/t/t0410-partial-clone.sh
@@ -3,6 +3,7 @@
 test_description='partial clone'
 
 . ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-terminal.sh
 
 # missing promisor objects cause repacks which write bitmaps to fail
 GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
@@ -689,6 +690,48 @@ test_expect_success 'lazy-fetch when accessing object not in the_repository' '
 	! grep "[?]$FILE_HASH" out
 '
 
+test_expect_success 'setup for promisor.quiet tests' '
+	rm -rf server &&
+	test_create_repo server &&
+	test_commit -C server foo &&
+	git -C server rm foo.t &&
+	git -C server commit -m remove &&
+	git -C server config uploadpack.allowanysha1inwant 1 &&
+	git -C server config uploadpack.allowfilter 1
+'
+
+test_expect_success TTY 'promisor.quiet=false shows progress messages' '
+	rm -rf repo &&
+	git clone --filter=blob:none "file://$(pwd)/server" repo &&
+	git -C repo config promisor.quiet "false" &&
+
+	test_terminal git -C repo cat-file -p foo:foo.t 2>err &&
+
+	# Ensure that progress messages are written
+	grep "Receiving objects" err
+'
+
+test_expect_success TTY 'promisor.quiet=true does not show progress messages' '
+	rm -rf repo &&
+	git clone --filter=blob:none "file://$(pwd)/server" repo &&
+	git -C repo config promisor.quiet "true" &&
+
+	test_terminal git -C repo cat-file -p foo:foo.t 2>err &&
+
+	# Ensure that no progress messages are written
+	! grep "Receiving objects" err
+'
+
+test_expect_success TTY 'promisor.quiet=unconfigured shows progress messages' '
+	rm -rf repo &&
+	git clone --filter=blob:none "file://$(pwd)/server" repo &&
+
+	test_terminal git -C repo cat-file -p foo:foo.t 2>err &&
+
+	# Ensure that progress messages are written
+	grep "Receiving objects" err
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd
 

-- 
2.45.1
Jeff King· May 25, 2024, 05:29 UTC · re: Tom Hughes · lore

Re: [PATCH] promisor-remote: add promisor.quiet configuration option

On Thu, May 23, 2024 at 02:19:26PM +0100, Tom Hughes wrote:
Show 6 quoted lines
> Add a configuration optione to allow output from the promisor
> fetching objects to be suppressed/
> 
> This allows us to stop commands like git blame being swamped
> with progress messages and gc notifications from the promisor
> when used in a partial clone.

I'm not at all opposed to providing a way to suppress this, but I feel like in the long run, the more fundamental issue is that git-blame kicks off a zillion fetches as it traverses. That's not only ugly but it's also horribly inefficient.

In an ideal world we'd queue all of the blobs we need, do a single fetch, and then compute the blame on the result. That's probably easier said than done, though we have done it in other spots (e.g., for checkout).

In terms of user experience, you can simulate it with something like:
  # fault in all of the necessary blobs in one batch
  git rev-list HEAD -- $file |
  git diff-tree --stdin --format= -r --diff-filter=d -m --raw -- $file |
  awk '{print $4}' |
  git -c fetch.negotiationAlgorithm=noop \
      fetch --no-tags --no-write-fetch-head --recurse-submodules=no \
      --filter=blob:none --stdin
  git blame $file

Obviously that command is horrid and not something users should have to care about. But if we had some way for blame to say "hey, I am traversing from X..Y, looking at these pathspecs", then our first lazy-fetch could try to grab all of them. And I think the same would be the case for "git log -p", and so on.

Doing a separate traversal isn't maximally efficient, but it might not be too bad in practice (and we could even do partial traversals to balance chunking versus responsiveness, though in the case of non-incremental blame we need everything before we generate an answer anyway).

But anyway, I bring it up here because I think once we reach that end state, it won't be as interesting to turn off the fetch progress.

-Peff
Tom Hughes· May 25, 2024, 10:29 UTC · re: Jeff King · lore

Re: [PATCH] promisor-remote: add promisor.quiet configuration option

On 25/05/2024 06:29, Jeff King wrote:
Show 13 quoted lines
> On Thu, May 23, 2024 at 02:19:26PM +0100, Tom Hughes wrote:
> 
>> Add a configuration optione to allow output from the promisor
>> fetching objects to be suppressed/
>>
>> This allows us to stop commands like git blame being swamped
>> with progress messages and gc notifications from the promisor
>> when used in a partial clone.
> 
> I'm not at all opposed to providing a way to suppress this, but I feel
> like in the long run, the more fundamental issue is that git-blame kicks
> off a zillion fetches as it traverses. That's not only ugly but it's
> also horribly inefficient.

This is true. One thing I found that makes things a lot more efficient if you're using ssh as the transport is to enable persistent multiplexing in .ssh/config with something like:

Host git.example.com
   ControlMaster auto
   ControlPath /run/user/%i/ssh/control.%C
   ControlPersist 1m
   SendEnv GIT_PROTOCOL

which avoids each fetch having to setup and authenticate a new ssh session.

> In an ideal world we'd queue all of the blobs we need, do a single
> fetch, and then compute the blame on the result. That's probably easier
> said than done, though we have done it in other spots (e.g., for
> checkout).
That would certainly be an excellent improvement, yes.
Tom
-- 
Tom Hughes (tom@compton.nu)
http://compton.nu/
Jeff King· May 29, 2024, 09:36 UTC · re: Tom Hughes · lore

Re: [PATCH] promisor-remote: add promisor.quiet configuration option

On Sat, May 25, 2024 at 11:29:13AM +0100, Tom Hughes wrote:
Show 17 quoted lines
> > I'm not at all opposed to providing a way to suppress this, but I feel
> > like in the long run, the more fundamental issue is that git-blame kicks
> > off a zillion fetches as it traverses. That's not only ugly but it's
> > also horribly inefficient.
> 
> This is true. One thing I found that makes things a lot more
> efficient if you're using ssh as the transport is to enable
> persistent multiplexing in .ssh/config with something like:
> 
> Host git.example.com
>   ControlMaster auto
>   ControlPath /run/user/%i/ssh/control.%C
>   ControlPersist 1m
>   SendEnv GIT_PROTOCOL
> 
> which avoids each fetch having to setup and authenticate a
> new ssh session.

Good point. That is sort of the opposite approach of my suggestion. That is, I was suggesting that git-blame batch everything to make a single efficient request. But if we could reduce the cost of making individual requests, then we wouldn't need to batch (which is quite a lot simpler).

The ssh session is going to be one source of latency and overhead. But just spawning the fetch and remote upload-pack are another (especially if you have to authenticate, and especially with the v2 protocol, which has an extra round-trip for capabilities upgrade).

If there was a long-running mode to git-fetch where it kept open a v2 session to the server and just said "hey, send me object X" and then "OK, now send me object Y" that would eliminate all of that overhead (and even for http, under the hood curl is good at keeping the session open between requests).

You'd still have some extra latency (while you're talking to the server, the local blame process is paused), but I suspect it would be a lot more tolerable.

And now your progress question is re-opened again. You might want a more succinct progress for something like blame that still does all of its fetching before generating output. E.g., you might want a single progress line with the current state (fetching or not), the count of fetched objects, the speed, and so on. And for something like "git log -p", where the progress would be interspersed with actual output, you might want to suppress it entirely.

So yeah, I have no real objection to what your patch is doing. Depending on how future work unfolds it might be more or less useful than it is now, but even in the worst case it probably won't be a bad thing to have in our toolbox.

-Peff

← back to recent threads