git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: non-smooth progress indication for git fsck and git gc

From
Junio C Hamano <gitster@pobox.com>
Date
Aug 16, 2018, 22:10 UTC
Message-ID
<xmqq36vedkud.fsf@gitster-ct.c.googlers.com>
In-Reply-To
<20180816200222.GA7422@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
Show 12 quoted lines
> On Thu, Aug 16, 2018 at 11:57:14AM -0400, Jeff King wrote:
>
>> The only way to solve that is to count bytes. We don't have a total byte
>> count in most cases, and it wouldn't always make sense (e.g., the
>> "Compressing objects" meter can show the same issue, but it's not really
>> putting through bytes in a linear way).  In some cases we do show
>> transmitted size and throughput, but that's just for network operations.
>> We could do the same for "gc" with the patch below. But usually
>> throughput isn't all that interesting for a filesystem write, because
>> bandwidth isn't the bottleneck.
>
> Just realized I forgot to include the patch. Here it is, for reference.
I've been wondering when you'd realize the omission ;-)
> Doing something similar for fsck would be quite a bit more invasive.

Yeah, on that codepath there is no streaming write passing through a single chokepoint you can count bytes X-<.

Show 62 quoted lines
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 80c880e9ad..e1130b959d 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -837,7 +837,7 @@ static void write_pack_file(void)
>  		if (pack_to_stdout)
>  			f = hashfd_throughput(1, "<stdout>", progress_state);
>  		else
> -			f = create_tmp_packfile(&pack_tmp_name);
> +			f = create_tmp_packfile(&pack_tmp_name, progress_state);
>  
>  		offset = write_pack_header(f, nr_remaining);
>  
> diff --git a/bulk-checkin.c b/bulk-checkin.c
> index 9f3b644811..0df45b8f55 100644
> --- a/bulk-checkin.c
> +++ b/bulk-checkin.c
> @@ -178,7 +178,7 @@ static void prepare_to_stream(struct bulk_checkin_state *state,
>  	if (!(flags & HASH_WRITE_OBJECT) || state->f)
>  		return;
>  
> -	state->f = create_tmp_packfile(&state->pack_tmp_name);
> +	state->f = create_tmp_packfile(&state->pack_tmp_name, NULL);
>  	reset_pack_idx_option(&state->pack_idx_opts);
>  
>  	/* Pretend we are going to write only one object */
> diff --git a/pack-write.c b/pack-write.c
> index a9d46bc03f..b72480b440 100644
> --- a/pack-write.c
> +++ b/pack-write.c
> @@ -334,14 +334,15 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,
>  	return n;
>  }
>  
> -struct hashfile *create_tmp_packfile(char **pack_tmp_name)
> +struct hashfile *create_tmp_packfile(char **pack_tmp_name,
> +				     struct progress *progress)
>  {
>  	struct strbuf tmpname = STRBUF_INIT;
>  	int fd;
>  
>  	fd = odb_mkstemp(&tmpname, "pack/tmp_pack_XXXXXX");
>  	*pack_tmp_name = strbuf_detach(&tmpname, NULL);
> -	return hashfd(fd, *pack_tmp_name);
> +	return hashfd_throughput(fd, *pack_tmp_name, progress);
>  }
>  
>  void finish_tmp_packfile(struct strbuf *name_buffer,
> diff --git a/pack.h b/pack.h
> index 34a9d458b4..c87628b093 100644
> --- a/pack.h
> +++ b/pack.h
> @@ -98,7 +98,8 @@ extern int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,
>  #define PH_ERROR_PROTOCOL	(-3)
>  extern int read_pack_header(int fd, struct pack_header *);
>  
> -extern struct hashfile *create_tmp_packfile(char **pack_tmp_name);
> +extern struct hashfile *create_tmp_packfile(char **pack_tmp_name,
> +					    struct progress *progress);
>  extern void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);
>  
>  #endif
Previous: Jeff KingNext: Ævar Arnfjörð Bjarmason
Message 7 of 25 in “non-smooth progress indication for git fsck and git gc”
  1. Ulrich WindlAug 16, 2018
  2. Duy NguyenAug 16, 2018
  3. Jeff KingAug 16, 2018
  4. Antw: Re: non-smooth progress indication for git fsck and git gcUlrich Windl, Aug 20, 2018
  5. Jeff KingAug 16, 2018
  6. Jeff KingAug 16, 2018
  7. Junio C HamanoAug 16, 2018
  8. Ævar Arnfjörð BjarmasonAug 16, 2018
  9. Jeff KingAug 16, 2018
  10. Jeff KingAug 16, 2018
  11. Duy NguyenAug 17, 2018
  12. Antw: Re: non-smooth progress indication for git fsck and git gcUlrich Windl, Aug 20, 2018
  13. Ævar Arnfjörð BjarmasonAug 20, 2018
  14. Ulrich WindlAug 20, 2018
  15. Jeff KingAug 21, 2018
  16. Ulrich WindlAug 21, 2018
  17. Duy NguyenAug 21, 2018
  18. Ævar Arnfjörð BjarmasonSep 1, 2018
  19. Ævar Arnfjörð BjarmasonSep 1, 2018
  20. Jeff KingSep 2, 2018
  21. Jeff KingSep 2, 2018
  22. Jeff KingSep 2, 2018
  23. Ævar Arnfjörð BjarmasonSep 3, 2018
  24. Jeff KingSep 7, 2018
  25. Junio C HamanoSep 4, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.