{"thread":{"id":"52930","subject":"[PATCH RESEND] remote-curl: show progress for fetches over dumb HTTP","startedAt":"2020-03-03T20:55:53Z","lastAt":"2020-03-05T07:03:57Z","messageCount":3,"participants":["René Scharfe","Junio C Hamano","Manish Devgan"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"392813","messageId":"e387d31d-afab-fe09-4e37-535a2650afca@web.de","threadId":"52930","inReplyTo":null,"subject":"[PATCH RESEND] remote-curl: show progress for fetches over dumb HTTP","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-03-03T20:55:34Z","receivedAt":"2020-03-03T20:55:53Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Fetching over dumb HTTP transport doesn't show any progress, even with\nthe option --progress.  If the connection is slow or there is a lot of\ndata to get then this can take a long time while the user is left to\nwonder if git got stuck.\n\nWe don't know the number of objects to fetch at the outset, but we can\ncount the ones we got.  Show an open-ended progress indicator based on\nthat number if the user asked for it.\n\nSigned-off-by: René Scharfe <l.s.r@web.de>\n---\nOriginal submission:\nhttps://lore.kernel.org/git/9ed26e7e-c19c-cdb2-0710-3b91bf31291b@web.de/\n\n remote-curl.c |  1 +\n walker.c      | 13 ++++++++++++-\n walker.h      |  1 +\n 3 files changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 8eb96152f5..e4cd321844 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -1026,6 +1026,7 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n\n \twalker = get_http_walker(url.buf);\n \twalker->get_verbosely = options.verbosity >= 3;\n+\twalker->get_progress = options.progress;\n \twalker->get_recover = 0;\n \tret = walker_fetch(walker, nr_heads, targets, NULL, NULL);\n \twalker_free(walker);\ndiff --git a/walker.c b/walker.c\nindex bb010f7a2b..4984bf8b3d 100644\n--- a/walker.c\n+++ b/walker.c\n@@ -8,6 +8,7 @@\n #include \"tag.h\"\n #include \"blob.h\"\n #include \"refs.h\"\n+#include \"progress.h\"\n\n static struct object_id current_commit_oid;\n\n@@ -162,6 +163,11 @@ static int process(struct walker *walker, struct object *obj)\n static int loop(struct walker *walker)\n {\n \tstruct object_list *elem;\n+\tstruct progress *progress = NULL;\n+\tuint64_t nr = 0;\n+\n+\tif (walker->get_progress)\n+\t\tprogress = start_delayed_progress(_(\"Fetching objects\"), 0);\n\n \twhile (process_queue) {\n \t\tstruct object *obj = process_queue->item;\n@@ -176,15 +182,20 @@ static int loop(struct walker *walker)\n \t\t */\n \t\tif (! (obj->flags & TO_SCAN)) {\n \t\t\tif (walker->fetch(walker, obj->oid.hash)) {\n+\t\t\t\tstop_progress(&progress);\n \t\t\t\treport_missing(obj);\n \t\t\t\treturn -1;\n \t\t\t}\n \t\t}\n \t\tif (!obj->type)\n \t\t\tparse_object(the_repository, &obj->oid);\n-\t\tif (process_object(walker, obj))\n+\t\tif (process_object(walker, obj)) {\n+\t\t\tstop_progress(&progress);\n \t\t\treturn -1;\n+\t\t}\n+\t\tdisplay_progress(progress, ++nr);\n \t}\n+\tstop_progress(&progress);\n \treturn 0;\n }\n\ndiff --git a/walker.h b/walker.h\nindex 6d8ae00e5b..d40b016bab 100644\n--- a/walker.h\n+++ b/walker.h\n@@ -10,6 +10,7 @@ struct walker {\n \tint (*fetch)(struct walker *, unsigned char *sha1);\n \tvoid (*cleanup)(struct walker *);\n \tint get_verbosely;\n+\tint get_progress;\n \tint get_recover;\n\n \tint corrupt_object_found;\n--\n2.25.1\n"},{"id":"392816","messageId":"xmqq1rq986bd.fsf@gitster-ct.c.googlers.com","threadId":"52930","inReplyTo":"e387d31d-afab-fe09-4e37-535a2650afca@web.de","subject":"Re: [PATCH RESEND] remote-curl: show progress for fetches over dumb HTTP","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-03T21:41:26Z","receivedAt":"2020-03-03T21:41:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe <l.s.r@web.de> writes:\n\n> Fetching over dumb HTTP transport doesn't show any progress, even with\n> the option --progress.  If the connection is slow or there is a lot of\n> data to get then this can take a long time while the user is left to\n> wonder if git got stuck.\n>\n> We don't know the number of objects to fetch at the outset, but we can\n> count the ones we got.  Show an open-ended progress indicator based on\n> that number if the user asked for it.\n>\n> Signed-off-by: René Scharfe <l.s.r@web.de>\n> ---\n> Original submission:\n> https://lore.kernel.org/git/9ed26e7e-c19c-cdb2-0710-3b91bf31291b@web.de/\n\nThanks.\n\n>\n>  remote-curl.c |  1 +\n>  walker.c      | 13 ++++++++++++-\n>  walker.h      |  1 +\n>  3 files changed, 14 insertions(+), 1 deletion(-)\n>\n> diff --git a/remote-curl.c b/remote-curl.c\n> index 8eb96152f5..e4cd321844 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -1026,6 +1026,7 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n>\n>  \twalker = get_http_walker(url.buf);\n>  \twalker->get_verbosely = options.verbosity >= 3;\n> +\twalker->get_progress = options.progress;\n>  \twalker->get_recover = 0;\n>  \tret = walker_fetch(walker, nr_heads, targets, NULL, NULL);\n>  \twalker_free(walker);\n> diff --git a/walker.c b/walker.c\n> index bb010f7a2b..4984bf8b3d 100644\n> --- a/walker.c\n> +++ b/walker.c\n> @@ -8,6 +8,7 @@\n>  #include \"tag.h\"\n>  #include \"blob.h\"\n>  #include \"refs.h\"\n> +#include \"progress.h\"\n>\n>  static struct object_id current_commit_oid;\n>\n> @@ -162,6 +163,11 @@ static int process(struct walker *walker, struct object *obj)\n>  static int loop(struct walker *walker)\n>  {\n>  \tstruct object_list *elem;\n> +\tstruct progress *progress = NULL;\n> +\tuint64_t nr = 0;\n> +\n> +\tif (walker->get_progress)\n> +\t\tprogress = start_delayed_progress(_(\"Fetching objects\"), 0);\n>\n>  \twhile (process_queue) {\n>  \t\tstruct object *obj = process_queue->item;\n> @@ -176,15 +182,20 @@ static int loop(struct walker *walker)\n>  \t\t */\n>  \t\tif (! (obj->flags & TO_SCAN)) {\n>  \t\t\tif (walker->fetch(walker, obj->oid.hash)) {\n> +\t\t\t\tstop_progress(&progress);\n>  \t\t\t\treport_missing(obj);\n>  \t\t\t\treturn -1;\n>  \t\t\t}\n>  \t\t}\n>  \t\tif (!obj->type)\n>  \t\t\tparse_object(the_repository, &obj->oid);\n> -\t\tif (process_object(walker, obj))\n> +\t\tif (process_object(walker, obj)) {\n> +\t\t\tstop_progress(&progress);\n>  \t\t\treturn -1;\n> +\t\t}\n> +\t\tdisplay_progress(progress, ++nr);\n>  \t}\n> +\tstop_progress(&progress);\n>  \treturn 0;\n>  }\n>\n> diff --git a/walker.h b/walker.h\n> index 6d8ae00e5b..d40b016bab 100644\n> --- a/walker.h\n> +++ b/walker.h\n> @@ -10,6 +10,7 @@ struct walker {\n>  \tint (*fetch)(struct walker *, unsigned char *sha1);\n>  \tvoid (*cleanup)(struct walker *);\n>  \tint get_verbosely;\n> +\tint get_progress;\n>  \tint get_recover;\n>\n>  \tint corrupt_object_found;\n> --\n> 2.25.1\n"},{"id":"392878","messageId":"CABVXwf60wvnpqu8x4c4H+3iGUaz_FbJPTv7UgNOHhz1+KfsnTA@mail.gmail.com","threadId":"52930","inReplyTo":"e387d31d-afab-fe09-4e37-535a2650afca@web.de","subject":"Re: [PATCH RESEND] remote-curl: show progress for fetches over dumb HTTP","fromName":"Manish Devgan","fromEmail":"manish.nsit8@gmail.com","sentAt":"2020-03-05T07:03:44Z","receivedAt":"2020-03-05T07:03:57Z","isPatch":true,"sender":{"key":"manish.nsit8@gmail.com","avatar":"https://gravatar.com/avatar/9248ecb21f4053d8d84f454840d7f0bfa8aa8aa9d73bd88d0f53a58cd2e7dc1b?d=mp&s=160"},"body":"On Wed, Mar 4, 2020 at 2:25 AM René Scharfe <l.s.r@web.de> wrote:\n>\n> Fetching over dumb HTTP transport doesn't show any progress, even with\n> the option --progress.  If the connection is slow or there is a lot of\n> data to get then this can take a long time while the user is left to\n> wonder if git got stuck.\n>\n> We don't know the number of objects to fetch at the outset, but we can\n> count the ones we got.  Show an open-ended progress indicator based on\n> that number if the user asked for it.\n>\n> Signed-off-by: René Scharfe <l.s.r@web.de>\n> ---\n> Original submission:\n> https://lore.kernel.org/git/9ed26e7e-c19c-cdb2-0710-3b91bf31291b@web.de/\n>\n>  remote-curl.c |  1 +\n>  walker.c      | 13 ++++++++++++-\n>  walker.h      |  1 +\n>  3 files changed, 14 insertions(+), 1 deletion(-)\n>\n> diff --git a/remote-curl.c b/remote-curl.c\n> index 8eb96152f5..e4cd321844 100644\n> --- a/remote-curl.c\n> +++ b/remote-curl.c\n> @@ -1026,6 +1026,7 @@ static int fetch_dumb(int nr_heads, struct ref **to_fetch)\n>\n>         walker = get_http_walker(url.buf);\n>         walker->get_verbosely = options.verbosity >= 3;\n> +       walker->get_progress = options.progress;\n>         walker->get_recover = 0;\n>         ret = walker_fetch(walker, nr_heads, targets, NULL, NULL);\n>         walker_free(walker);\n> diff --git a/walker.c b/walker.c\n> index bb010f7a2b..4984bf8b3d 100644\n> --- a/walker.c\n> +++ b/walker.c\n> @@ -8,6 +8,7 @@\n>  #include \"tag.h\"\n>  #include \"blob.h\"\n>  #include \"refs.h\"\n> +#include \"progress.h\"\n>\n>  static struct object_id current_commit_oid;\n>\n> @@ -162,6 +163,11 @@ static int process(struct walker *walker, struct object *obj)\n>  static int loop(struct walker *walker)\n>  {\n>         struct object_list *elem;\n> +       struct progress *progress = NULL;\n> +       uint64_t nr = 0;\n> +\n> +       if (walker->get_progress)\n> +               progress = start_delayed_progress(_(\"Fetching objects\"), 0);\n>\n>         while (process_queue) {\n>                 struct object *obj = process_queue->item;\n> @@ -176,15 +182,20 @@ static int loop(struct walker *walker)\n>                  */\n>                 if (! (obj->flags & TO_SCAN)) {\n>                         if (walker->fetch(walker, obj->oid.hash)) {\n> +                               stop_progress(&progress);\n>                                 report_missing(obj);\n>                                 return -1;\n>                         }\n>                 }\n>                 if (!obj->type)\n>                         parse_object(the_repository, &obj->oid);\n> -               if (process_object(walker, obj))\n> +               if (process_object(walker, obj)) {\n> +                       stop_progress(&progress);\n>                         return -1;\n> +               }\n> +               display_progress(progress, ++nr);\n>         }\n> +       stop_progress(&progress);\n>         return 0;\n>  }\n>\n> diff --git a/walker.h b/walker.h\n> index 6d8ae00e5b..d40b016bab 100644\n> --- a/walker.h\n> +++ b/walker.h\n> @@ -10,6 +10,7 @@ struct walker {\n>         int (*fetch)(struct walker *, unsigned char *sha1);\n>         void (*cleanup)(struct walker *);\n>         int get_verbosely;\n> +       int get_progress;\n>         int get_recover;\n>\n>         int corrupt_object_found;\n> --\n> 2.25.1\n\n\nThanks. This works for me.\n\nRegards\nManish Devgan\n"}]}