patchfetch: show progress for packfile uri downloads

From: Albert Cui <albertqcui@gmail.com>

Git appears to hang when downloading packfiles as this part of the fetch is silent, causing user confusion. This change implements progress for the number of packfiles downloaded; a progress display for bytes would involve deeper changes at the http-fetch layer instead of fetch-pack, the caller.

Signed-off-by: Albert Cui <albertqcui@gmail.com>
---
    fetch: show progress for packfile uri downloads
    
    Git appears to hang when downloading packfiles as this part of the fetch
    is silent, causing user confusion. This change implements progress for
    the number of packfiles downloaded; a progress display for bytes would
    involve deeper changes at the http-fetch layer instead of fetch-pack,
    the caller.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-907%2Falbertcui%2Fprogress-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-907/albertcui/progress-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/907

 fetch-pack.c           | 8 ++++++++
 t/t5702-protocol-v2.sh | 5 ++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

On Wed, Mar 17, 2021 at 07:03:13PM +0000, Albert Cui via GitGitGadget wrote:

I think this is an improvement, but I agree that a real byte display would be much better. I actually worked on this a long time ago for a very similar feature that we never quite pushed over the finish line. See patches 11 and 12 here:

  https://lore.kernel.org/git/20111110074330.GA27925@sigill.intra.peff.net/

(it might need some of the earlier refactoring, too, I'm not sure; and quite likely will need forward-porting as it has been 10 years).

-Peff

"Albert Cui via GitGitGadget" <gitgitgadget@gmail.com> writes:

... "hence we do not do so in this patch"?  

That's probably a very sensible way to go.

I expect that http-fetch will in the longer term become a mere fallback default used by those who do not have anything better. Because we are not in the business of writing a performant HTTP downloader, we would be better off if we make it easy to plug an external HTTP downloader other people write in to this codepath.

OK, so we plan to count from 0 up to .nr; and the message is made localizable. Good.

		display_progress(packfile_uri_progress, i + 1);

Some existing tests use GIT_PROGRESS_DELAY to protect against an operation that is too quick to complete. Don't we need to do the same? If not, then perhaps we need to allow delaying the progress meter we add with this patch for "too quick" case, perhaps?

Also, I am not sure with all the terminal control junk, 'grep' should be expected to reliably pick this substring in the output. Are we expecting any other output to the standard error stream? Some tests in t5318 seem to just see if the output is non-empty, and I am wondering if that is an approach more appropriate here (not rhetorical---I simply do not know the answer).

From: Albert Cui <albertqcui@gmail.com>

Git appears to hang when downloading packfiles as this part of the fetch is silent, causing user confusion. This change implements progress for the number of packfiles downloaded; a progress display for bytes would involve deeper changes at the http-fetch layer instead of fetch-pack, the caller, so we do not do that in this patch.

Signed-off-by: Albert Cui <albertqcui@gmail.com>
---
    fetch: show progress for packfile uri downloads
    
    Git appears to hang when downloading packfiles as this part of the fetch
    is silent, causing user confusion. This change implements progress for
    the number of packfiles downloaded; a progress display for bytes would
    involve deeper changes at the http-fetch layer instead of fetch-pack,
    the caller.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-907%2Falbertcui%2Fprogress-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-907/albertcui/progress-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/907

Range-diff vs v1:

 1:  d6cf8c195111 ! 1:  aedb1b33d0c4 fetch: show progress for packfile uri downloads
     @@ Commit message
          fetch is silent, causing user confusion. This change implements
          progress for the number of packfiles downloaded; a progress display
          for bytes would involve deeper changes at the http-fetch layer
     -    instead of fetch-pack, the caller.
     +    instead of fetch-pack, the caller, so we do not do that in this
     +    patch.
      
          Signed-off-by: Albert Cui <albertqcui@gmail.com>
      
     @@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
       		const char *uri = packfile_uris.items[i].string +
       			the_hash_algo->hexsz + 1;
       
     -+		display_progress(packfile_uri_progress, i+1);
     ++		display_progress(packfile_uri_progress, i + 1);
       		strvec_push(&cmd.args, "http-fetch");
       		strvec_pushf(&cmd.args, "--packfile=%.*s",
       			     (int) the_hash_algo->hexsz,
     @@ fetch-pack.c: static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
      
       ## t/t5702-protocol-v2.sh ##
      @@ t/t5702-protocol-v2.sh: test_expect_success 'part of packfile response provided as URI' '
     - 	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
     + 	configure_exclusion "$P" my-blob >h &&
     + 	configure_exclusion "$P" other-blob >h2 &&
     + 
     +-	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
     ++	GIT_PROGRESS_DELAY=0 GIT_TRACE=1 GIT_TRACE2_EVENT=1 \
     ++	GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
       	git -c protocol.version=2 \
       		-c fetch.uriprotocols=http,https \
      -		clone "$HTTPD_URL/smart/http_parent" http_child &&

 fetch-pack.c           | 8 ++++++++
 t/t5702-protocol-v2.sh | 8 ++++++--
 2 files changed, 14 insertions(+), 2 deletions(-)

"Albert Cui via GitGitGadget" <gitgitgadget@gmail.com> writes:

... meaning, hopefully later we'd hook into transport->progress and implement the byte-level progress display down there? And when that happens, we'd remove this file-level progress as it would be too confusing to have both at the same time?

Is this start_progress() call a way to unconditionally enable the progress display? How does it interact with transport->progress that is driven by transport_set_verbosity(), which in turn is called by builtin/fetch.c and friends? If it doesn't, shouldn't this codepath pay attention to the transport->progress and enable the progress meter only when it is enabled (i.e. the stderr going to a terminal, or --progress explicitly being asked)?

Hi,

Albert Cui wrote:

This is something that came up at the last in-person Git Contributor Summit; I'm glad to see it being taken care of.

It seems to be more idiomatic to initialize this to NULL.

That way, we can respect a --quiet option by making this remain NULL when progress is not enabled:

	if (!args->quiet && !args->no_progress)
		packfile_uri_progress = ...;

[...]

This puts the trace in stderr mixed with other output. Would it make sense to put it in a separate file, like this?

	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" \
	GIT_PROGRESS_DELAY=0 GIT_TRACE2_EVENT="$(pwd)/trace2" \
	GIT_TEST_SIDEBAND_ALL=1 \
	git -c [etc]

[...]

That way, this "grep" could check the trace2 file which would contain output intended for machines, and we wouldn't have to worry e.g. about ANSII control codes potentially affecting the output around the space some day in the progress output intended for a terminal.

With whatever subset of the changes described above make sense, this is
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Thanks.

On Wed, Mar 17 2021, Albert Cui via GitGitGadget wrote:

This can just be "grep"