{"thread":{"id":"49299","subject":"[PATCH] status: show progress bar if refreshing the index takes too long","startedAt":"2018-09-07T15:51:44Z","lastAt":"2018-09-15T17:56:22Z","messageCount":5,"participants":["Nguyễn Thái Ngọc Duy","Eric Sunshine","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"357630","messageId":"20180907155133.27737-1-pclouds@gmail.com","threadId":"49299","inReplyTo":null,"subject":"[PATCH] status: show progress bar if refreshing the index takes too long","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-09-07T15:51:33Z","receivedAt":"2018-09-07T15:51:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Refreshing the index is usually very fast, but it can still take a\nlong time sometimes. Cold cache is one, or something else silly (*).\nIn this case, it's good to show something to let the user know \"git\nstatus\" is not hanging, it's just busy doing something.\n\n(*) I got called by my colleague because her \"git status\" took very\n    long and looked pretty much like hanging. After a bit of strace,\n    it looks to me that git was trying to rehash every single file,\n    and this was a big repository. This process could take minutes.\n\n    In this case, I think it was probably because she copied this\n    repository to a new place and stat data did not match anymore. So\n    git fell back to hashing.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n I need to get this out of my head before I forget. I obviously think\n this is a good idea and could be done in more places, even just to\n deal with cold cache. The hint about \"git status -uno\" for example,\n could be accompanied by a progress bar for scanning for untracked\n files...\n\n Another note about rehashing files as part of refresh. We probably\n could do better by hashing in parallel. Or perhaps not because having\n a big lock around object database pretty much kills performance, and\n if I remember correctly none of my core was 100% consumed (i.e. CPU\n bottleneck as an indication that multithread is a good idea...)\n\n Anyway that's it! Weekend after a long week! I'll read mails and\n respond tomorrow.\n\n builtin/am.c     |  2 +-\n builtin/commit.c |  6 ++++--\n cache.h          |  7 +++++--\n preload-index.c  | 44 +++++++++++++++++++++++++++++++++++++++-----\n read-cache.c     | 10 ++++++++++\n sequencer.c      |  2 +-\n 6 files changed, 60 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 5e866d17c7..22a93cfef3 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -2324,7 +2324,7 @@ int cmd_am(int argc, const char **argv, const char *prefix)\n \t/* Ensure a valid committer ident can be constructed */\n \tgit_committer_info(IDENT_STRICT);\n \n-\tif (read_index_preload(&the_index, NULL) < 0)\n+\tif (read_index_preload(&the_index, NULL, 0) < 0)\n \t\tdie(_(\"failed to read the index\"));\n \n \tif (in_progress) {\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 0d9828e29e..eaf639ece6 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1355,8 +1355,10 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \t\t       PATHSPEC_PREFER_FULL,\n \t\t       prefix, argv);\n \n-\tread_cache_preload(&s.pathspec);\n-\trefresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, &s.pathspec, NULL, NULL);\n+\tread_index_preload(&the_index, &s.pathspec, REFRESH_PROGRESS);\n+\trefresh_index(&the_index,\n+\t\t      REFRESH_QUIET|REFRESH_UNMERGED|REFRESH_PROGRESS,\n+\t\t      &s.pathspec, NULL, NULL);\n \n \tif (use_optional_locks())\n \t\tfd = hold_locked_index(&index_lock, 0);\ndiff --git a/cache.h b/cache.h\nindex 4d014541ab..35da02be90 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -410,7 +410,7 @@ void validate_cache_entries(const struct index_state *istate);\n \n #define read_cache() read_index(&the_index)\n #define read_cache_from(path) read_index_from(&the_index, (path), (get_git_dir()))\n-#define read_cache_preload(pathspec) read_index_preload(&the_index, (pathspec))\n+#define read_cache_preload(pathspec) read_index_preload(&the_index, (pathspec), 0)\n #define is_cache_unborn() is_index_unborn(&the_index)\n #define read_cache_unmerged() read_index_unmerged(&the_index)\n #define discard_cache() discard_index(&the_index)\n@@ -659,7 +659,9 @@ extern int daemonize(void);\n /* Initialize and use the cache information */\n struct lock_file;\n extern int read_index(struct index_state *);\n-extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);\n+extern int read_index_preload(struct index_state *,\n+\t\t\t      const struct pathspec *pathspec,\n+\t\t\t      unsigned int refresh_flags);\n extern int do_read_index(struct index_state *istate, const char *path,\n \t\t\t int must_exist); /* for testting only! */\n extern int read_index_from(struct index_state *, const char *path,\n@@ -814,6 +816,7 @@ extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n #define REFRESH_IGNORE_MISSING\t0x0008\t/* ignore non-existent */\n #define REFRESH_IGNORE_SUBMODULES\t0x0010\t/* ignore submodules */\n #define REFRESH_IN_PORCELAIN\t0x0020\t/* user friendly output, not \"needs update\" */\n+#define REFRESH_PROGRESS\t0x0040  /* show progress bar if stderr is tty */\n extern int refresh_index(struct index_state *, unsigned int flags, const struct pathspec *pathspec, char *seen, const char *header_msg);\n extern struct cache_entry *refresh_cache_entry(struct index_state *, struct cache_entry *, unsigned int);\n \ndiff --git a/preload-index.c b/preload-index.c\nindex 71cd2437a3..bf7dbed779 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -5,10 +5,12 @@\n #include \"pathspec.h\"\n #include \"dir.h\"\n #include \"fsmonitor.h\"\n+#include \"progress.h\"\n \n #ifdef NO_PTHREADS\n static void preload_index(struct index_state *index,\n-\t\t\t  const struct pathspec *pathspec)\n+\t\t\t  const struct pathspec *pathspec,\n+\t\t\t  unsigned int refresh_flags)\n {\n \t; /* nothing */\n }\n@@ -25,16 +27,23 @@ static void preload_index(struct index_state *index,\n #define MAX_PARALLEL (20)\n #define THREAD_COST (500)\n \n+struct progress_data {\n+\tunsigned long n;\n+\tstruct progress *progress;\n+\tpthread_mutex_t mutex;\n+};\n+\n struct thread_data {\n \tpthread_t pthread;\n \tstruct index_state *index;\n \tstruct pathspec pathspec;\n+\tstruct progress_data *progress;\n \tint offset, nr;\n };\n \n static void *preload_thread(void *_data)\n {\n-\tint nr;\n+\tint nr, last_nr;\n \tstruct thread_data *p = _data;\n \tstruct index_state *index = p->index;\n \tstruct cache_entry **cep = index->cache + p->offset;\n@@ -43,6 +52,7 @@ static void *preload_thread(void *_data)\n \tnr = p->nr;\n \tif (nr + p->offset > index->cache_nr)\n \t\tnr = index->cache_nr - p->offset;\n+\tlast_nr = nr;\n \n \tdo {\n \t\tstruct cache_entry *ce = *cep++;\n@@ -58,6 +68,15 @@ static void *preload_thread(void *_data)\n \t\t\tcontinue;\n \t\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n \t\t\tcontinue;\n+\t\tif (p->progress && !(nr & 31)) {\n+\t\t\tstruct progress_data *pd = p->progress;\n+\n+\t\t\tpthread_mutex_lock(&pd->mutex);\n+\t\t\tpd->n += last_nr - nr;\n+\t\t\tdisplay_progress(pd->progress, pd->n);\n+\t\t\tpthread_mutex_unlock(&pd->mutex);\n+\t\t\tlast_nr = nr;\n+\t\t}\n \t\tif (!ce_path_match(index, ce, &p->pathspec, NULL))\n \t\t\tcontinue;\n \t\tif (threaded_has_symlink_leading_path(&cache, ce->name, ce_namelen(ce)))\n@@ -74,11 +93,13 @@ static void *preload_thread(void *_data)\n }\n \n static void preload_index(struct index_state *index,\n-\t\t\t  const struct pathspec *pathspec)\n+\t\t\t  const struct pathspec *pathspec,\n+\t\t\t  unsigned int refresh_flags)\n {\n \tint threads, i, work, offset;\n \tstruct thread_data data[MAX_PARALLEL];\n \tuint64_t start = getnanotime();\n+\tstruct progress_data pd;\n \n \tif (!core_preload_index)\n \t\treturn;\n@@ -93,6 +114,13 @@ static void preload_index(struct index_state *index,\n \toffset = 0;\n \twork = DIV_ROUND_UP(index->cache_nr, threads);\n \tmemset(&data, 0, sizeof(data));\n+\n+\tmemset(&pd, 0, sizeof(pd));\n+\tif (refresh_flags & REFRESH_PROGRESS && isatty(2)) {\n+\t\tpd.progress = start_delayed_progress(_(\"Refreshing index\"), index->cache_nr);\n+\t\tpthread_mutex_init(&pd.mutex, NULL);\n+\t}\n+\n \tfor (i = 0; i < threads; i++) {\n \t\tstruct thread_data *p = data+i;\n \t\tp->index = index;\n@@ -100,6 +128,8 @@ static void preload_index(struct index_state *index,\n \t\t\tcopy_pathspec(&p->pathspec, pathspec);\n \t\tp->offset = offset;\n \t\tp->nr = work;\n+\t\tif (pd.progress)\n+\t\t\tp->progress = &pd;\n \t\toffset += work;\n \t\tif (pthread_create(&p->pthread, NULL, preload_thread, p))\n \t\t\tdie(\"unable to create threaded lstat\");\n@@ -109,15 +139,19 @@ static void preload_index(struct index_state *index,\n \t\tif (pthread_join(p->pthread, NULL))\n \t\t\tdie(\"unable to join threaded lstat\");\n \t}\n+\tif (pd.progress)\n+\t\tstop_progress(&pd.progress);\n+\n \ttrace_performance_since(start, \"preload index\");\n }\n #endif\n \n int read_index_preload(struct index_state *index,\n-\t\t       const struct pathspec *pathspec)\n+\t\t       const struct pathspec *pathspec,\n+\t\t       unsigned int refresh_flags)\n {\n \tint retval = read_index(index);\n \n-\tpreload_index(index, pathspec);\n+\tpreload_index(index, pathspec, refresh_flags);\n \treturn retval;\n }\ndiff --git a/read-cache.c b/read-cache.c\nindex 7b1354d759..e8725a5162 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -23,6 +23,7 @@\n #include \"split-index.h\"\n #include \"utf8.h\"\n #include \"fsmonitor.h\"\n+#include \"progress.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -1477,6 +1478,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tconst char *added_fmt;\n \tconst char *unmerged_fmt;\n \tuint64_t start = getnanotime();\n+\tstruct progress *progress = NULL;\n+\n+\tif (flags & REFRESH_PROGRESS && isatty(2))\n+\t\tprogress = start_delayed_progress(_(\"Refresh index\"),\n+\t\t\t\t\t\t  istate->cache_nr);\n \n \tmodified_fmt = (in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\");\n \tdeleted_fmt = (in_porcelain ? \"D\\t%s\\n\" : \"%s: needs update\\n\");\n@@ -1516,6 +1522,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tnew_entry = refresh_cache_ent(istate, ce, options, &cache_errno, &changed);\n \t\tif (new_entry == ce)\n \t\t\tcontinue;\n+\t\tif (progress)\n+\t\t\tdisplay_progress(progress, i);\n \t\tif (!new_entry) {\n \t\t\tconst char *fmt;\n \n@@ -1547,6 +1555,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \n \t\treplace_index_entry(istate, i, new_entry);\n \t}\n+\tif (progress)\n+\t\tstop_progress(&progress);\n \ttrace_performance_since(start, \"refresh index\");\n \treturn has_errors;\n }\ndiff --git a/sequencer.c b/sequencer.c\nindex dc2c58d464..e0cd17df70 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1909,7 +1909,7 @@ static int read_and_refresh_cache(struct replay_opts *opts)\n {\n \tstruct lock_file index_lock = LOCK_INIT;\n \tint index_fd = hold_locked_index(&index_lock, 0);\n-\tif (read_index_preload(&the_index, NULL) < 0) {\n+\tif (read_index_preload(&the_index, NULL, 0) < 0) {\n \t\trollback_lock_file(&index_lock);\n \t\treturn error(_(\"git %s: failed to read the index\"),\n \t\t\t_(action_name(opts)));\n-- \n2.19.0.rc0.337.ge906d732e7\n\n"},{"id":"357638","messageId":"CAPig+cQVCUyt5NvcSCAO63b1aO8HSc5AobCRKP5mp9jT_LYuJQ@mail.gmail.com","threadId":"49299","inReplyTo":"20180907155133.27737-1-pclouds@gmail.com","subject":"Re: [PATCH] status: show progress bar if refreshing the index takes too long","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-09-07T17:38:23Z","receivedAt":"2018-09-07T17:38:45Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Sep 7, 2018 at 11:51 AM Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n> Refreshing the index is usually very fast, but it can still take a\n> long time sometimes. Cold cache is one, or something else silly (*).\n> In this case, it's good to show something to let the user know \"git\n> status\" is not hanging, it's just busy doing something.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n> ---\n> diff --git a/read-cache.c b/read-cache.c\n> @@ -1516,6 +1522,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> +               if (progress)\n> +                       display_progress(progress, i);\n> @@ -1547,6 +1555,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> +       if (progress)\n> +               stop_progress(&progress);\n\nNit: Both display_progress() and stop_progress() behave sanely when\n'progress' is NULL, so no need for the conditional.\n"},{"id":"357654","messageId":"796edb6b-74ec-35dc-4bae-7a817f8fa9b5@gmail.com","threadId":"49299","inReplyTo":"CAPig+cQVCUyt5NvcSCAO63b1aO8HSc5AobCRKP5mp9jT_LYuJQ@mail.gmail.com","subject":"Re: [PATCH] status: show progress bar if refreshing the index takes too long","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-07T20:29:45Z","receivedAt":"2018-09-07T20:29:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\nOn 9/7/2018 1:38 PM, Eric Sunshine wrote:\n> On Fri, Sep 7, 2018 at 11:51 AM Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n>> Refreshing the index is usually very fast, but it can still take a\n>> long time sometimes. Cold cache is one, or something else silly (*).\n>> In this case, it's good to show something to let the user know \"git\n>> status\" is not hanging, it's just busy doing something.\n>>\n>> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n>> ---\n>> diff --git a/read-cache.c b/read-cache.c\n>> @@ -1516,6 +1522,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n>> +               if (progress)\n>> +                       display_progress(progress, i);\n>> @@ -1547,6 +1555,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n>> +       if (progress)\n>> +               stop_progress(&progress);\n> Nit: Both display_progress() and stop_progress() behave sanely when\n> 'progress' is NULL, so no need for the conditional.\n\nDon't forget this one in preload-index.c:preload_index():\n\n+\tif (pd.progress)\n+\t\tstop_progress(&pd.progress);\n\nI found this extra one by creating the following rules in a Coccinelle script:\n\n@@\nexpression e;\n@@\n- if (e) { stop_progress(&e); }\n+ stop_progress(&e);\n\n@@\nexpression e;\nexpression i;\n@@\n- if (e) { display_progress(e, i); }\n+ display_progress(e, i);\n\n\nNot sure if we want to put these in a .cocci script or not.\n\nThanks,\n-Stolee\n\n"},{"id":"357656","messageId":"CAPig+cRvbvH4cZZhWTf2xHrJuD8n75bdJNX2BMHgSCveppOC+w@mail.gmail.com","threadId":"49299","inReplyTo":"796edb6b-74ec-35dc-4bae-7a817f8fa9b5@gmail.com","subject":"Re: [PATCH] status: show progress bar if refreshing the index takes too long","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-09-07T21:10:00Z","receivedAt":"2018-09-07T21:10:14Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Sep 7, 2018 at 4:29 PM Derrick Stolee <stolee@gmail.com> wrote:\n> On 9/7/2018 1:38 PM, Eric Sunshine wrote:\n> > On Fri, Sep 7, 2018 at 11:51 AM Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n> >> +               if (progress)\n> >> +                       display_progress(progress, i);\n> >> +       if (progress)\n> >> +               stop_progress(&progress);\n> > Nit: Both display_progress() and stop_progress() behave sanely when\n> > 'progress' is NULL, so no need for the conditional.\n>\n> Don't forget this one in preload-index.c:preload_index():\n> +       if (pd.progress)\n> +               stop_progress(&pd.progress);\n>\n> I found this extra one by creating the following rules in a Coccinelle script:\n> - if (e) { stop_progress(&e); }\n> - if (e) { display_progress(e, i); }\n> Not sure if we want to put these in a .cocci script or not.\n\nIf so, we'd want to match display_throughput() and stop_progress_msg(), as well.\n"},{"id":"358217","messageId":"20180915175604.27801-1-pclouds@gmail.com","threadId":"49299","inReplyTo":"20180907155133.27737-1-pclouds@gmail.com","subject":"[PATCH v2] status: show progress bar if refreshing the index takes too long","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-09-15T17:56:04Z","receivedAt":"2018-09-15T17:56:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Refreshing the index is usually very fast, but it can still take a\nlong time sometimes. Cold cache is one. Or copying a repo to a new\nplace (*). It's good to show something to let the user know \"git\nstatus\" is not hanging, it's just busy doing something.\n\n(*) In this case, all stat info in the index becomes invalid and git\n    falls back to rehashing all file content to see if there's any\n    difference between updating stat info in the index. This is quite\n    expensive. Even with a repo as small as git.git, it takes 3\n    seconds.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n v2 changes\n\n  - call stop_progress() without checking if the progress_data object\n    is NULL.\n\n  - disable progress bar when outputting porcelain formats\n\n builtin/am.c     |  2 +-\n builtin/commit.c | 10 ++++++++--\n cache.h          |  7 +++++--\n preload-index.c  | 50 +++++++++++++++++++++++++++++++++++++++++++-----\n read-cache.c     | 12 ++++++++++++\n sequencer.c      |  2 +-\n 6 files changed, 72 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 5e866d17c7..22a93cfef3 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -2324,7 +2324,7 @@ int cmd_am(int argc, const char **argv, const char *prefix)\n \t/* Ensure a valid committer ident can be constructed */\n \tgit_committer_info(IDENT_STRICT);\n \n-\tif (read_index_preload(&the_index, NULL) < 0)\n+\tif (read_index_preload(&the_index, NULL, 0) < 0)\n \t\tdie(_(\"failed to read the index\"));\n \n \tif (in_progress) {\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 0d9828e29e..7523463066 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1295,6 +1295,7 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tstatic int no_renames = -1;\n \tstatic const char *rename_score_arg = (const char *)-1;\n \tstatic struct wt_status s;\n+\tunsigned int progress_flag = 0;\n \tint fd;\n \tstruct object_id oid;\n \tstatic struct option builtin_status_options[] = {\n@@ -1355,8 +1356,13 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \t\t       PATHSPEC_PREFER_FULL,\n \t\t       prefix, argv);\n \n-\tread_cache_preload(&s.pathspec);\n-\trefresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, &s.pathspec, NULL, NULL);\n+\tif (status_format != STATUS_FORMAT_PORCELAIN &&\n+\t    status_format != STATUS_FORMAT_PORCELAIN_V2)\n+\t\tprogress_flag = REFRESH_PROGRESS;\n+\tread_index_preload(&the_index, &s.pathspec, progress_flag);\n+\trefresh_index(&the_index,\n+\t\t      REFRESH_QUIET|REFRESH_UNMERGED|progress_flag,\n+\t\t      &s.pathspec, NULL, NULL);\n \n \tif (use_optional_locks())\n \t\tfd = hold_locked_index(&index_lock, 0);\ndiff --git a/cache.h b/cache.h\nindex 4d014541ab..35da02be90 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -410,7 +410,7 @@ void validate_cache_entries(const struct index_state *istate);\n \n #define read_cache() read_index(&the_index)\n #define read_cache_from(path) read_index_from(&the_index, (path), (get_git_dir()))\n-#define read_cache_preload(pathspec) read_index_preload(&the_index, (pathspec))\n+#define read_cache_preload(pathspec) read_index_preload(&the_index, (pathspec), 0)\n #define is_cache_unborn() is_index_unborn(&the_index)\n #define read_cache_unmerged() read_index_unmerged(&the_index)\n #define discard_cache() discard_index(&the_index)\n@@ -659,7 +659,9 @@ extern int daemonize(void);\n /* Initialize and use the cache information */\n struct lock_file;\n extern int read_index(struct index_state *);\n-extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);\n+extern int read_index_preload(struct index_state *,\n+\t\t\t      const struct pathspec *pathspec,\n+\t\t\t      unsigned int refresh_flags);\n extern int do_read_index(struct index_state *istate, const char *path,\n \t\t\t int must_exist); /* for testting only! */\n extern int read_index_from(struct index_state *, const char *path,\n@@ -814,6 +816,7 @@ extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n #define REFRESH_IGNORE_MISSING\t0x0008\t/* ignore non-existent */\n #define REFRESH_IGNORE_SUBMODULES\t0x0010\t/* ignore submodules */\n #define REFRESH_IN_PORCELAIN\t0x0020\t/* user friendly output, not \"needs update\" */\n+#define REFRESH_PROGRESS\t0x0040  /* show progress bar if stderr is tty */\n extern int refresh_index(struct index_state *, unsigned int flags, const struct pathspec *pathspec, char *seen, const char *header_msg);\n extern struct cache_entry *refresh_cache_entry(struct index_state *, struct cache_entry *, unsigned int);\n \ndiff --git a/preload-index.c b/preload-index.c\nindex 71cd2437a3..2541b307e8 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -5,10 +5,12 @@\n #include \"pathspec.h\"\n #include \"dir.h\"\n #include \"fsmonitor.h\"\n+#include \"progress.h\"\n \n #ifdef NO_PTHREADS\n static void preload_index(struct index_state *index,\n-\t\t\t  const struct pathspec *pathspec)\n+\t\t\t  const struct pathspec *pathspec,\n+\t\t\t  unsigned int refresh_flags)\n {\n \t; /* nothing */\n }\n@@ -25,16 +27,23 @@ static void preload_index(struct index_state *index,\n #define MAX_PARALLEL (20)\n #define THREAD_COST (500)\n \n+struct progress_data {\n+\tunsigned long n;\n+\tstruct progress *progress;\n+\tpthread_mutex_t mutex;\n+};\n+\n struct thread_data {\n \tpthread_t pthread;\n \tstruct index_state *index;\n \tstruct pathspec pathspec;\n+\tstruct progress_data *progress;\n \tint offset, nr;\n };\n \n static void *preload_thread(void *_data)\n {\n-\tint nr;\n+\tint nr, last_nr;\n \tstruct thread_data *p = _data;\n \tstruct index_state *index = p->index;\n \tstruct cache_entry **cep = index->cache + p->offset;\n@@ -43,6 +52,7 @@ static void *preload_thread(void *_data)\n \tnr = p->nr;\n \tif (nr + p->offset > index->cache_nr)\n \t\tnr = index->cache_nr - p->offset;\n+\tlast_nr = nr;\n \n \tdo {\n \t\tstruct cache_entry *ce = *cep++;\n@@ -58,6 +68,15 @@ static void *preload_thread(void *_data)\n \t\t\tcontinue;\n \t\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n \t\t\tcontinue;\n+\t\tif (p->progress && !(nr & 31)) {\n+\t\t\tstruct progress_data *pd = p->progress;\n+\n+\t\t\tpthread_mutex_lock(&pd->mutex);\n+\t\t\tpd->n += last_nr - nr;\n+\t\t\tdisplay_progress(pd->progress, pd->n);\n+\t\t\tpthread_mutex_unlock(&pd->mutex);\n+\t\t\tlast_nr = nr;\n+\t\t}\n \t\tif (!ce_path_match(index, ce, &p->pathspec, NULL))\n \t\t\tcontinue;\n \t\tif (threaded_has_symlink_leading_path(&cache, ce->name, ce_namelen(ce)))\n@@ -69,16 +88,25 @@ static void *preload_thread(void *_data)\n \t\tce_mark_uptodate(ce);\n \t\tmark_fsmonitor_valid(ce);\n \t} while (--nr > 0);\n+\tif (p->progress) {\n+\t\tstruct progress_data *pd = p->progress;\n+\n+\t\tpthread_mutex_lock(&pd->mutex);\n+\t\tdisplay_progress(pd->progress, pd->n + last_nr);\n+\t\tpthread_mutex_unlock(&pd->mutex);\n+\t}\n \tcache_def_clear(&cache);\n \treturn NULL;\n }\n \n static void preload_index(struct index_state *index,\n-\t\t\t  const struct pathspec *pathspec)\n+\t\t\t  const struct pathspec *pathspec,\n+\t\t\t  unsigned int refresh_flags)\n {\n \tint threads, i, work, offset;\n \tstruct thread_data data[MAX_PARALLEL];\n \tuint64_t start = getnanotime();\n+\tstruct progress_data pd;\n \n \tif (!core_preload_index)\n \t\treturn;\n@@ -93,6 +121,13 @@ static void preload_index(struct index_state *index,\n \toffset = 0;\n \twork = DIV_ROUND_UP(index->cache_nr, threads);\n \tmemset(&data, 0, sizeof(data));\n+\n+\tmemset(&pd, 0, sizeof(pd));\n+\tif (refresh_flags & REFRESH_PROGRESS && isatty(2)) {\n+\t\tpd.progress = start_delayed_progress(_(\"Refreshing index\"), index->cache_nr);\n+\t\tpthread_mutex_init(&pd.mutex, NULL);\n+\t}\n+\n \tfor (i = 0; i < threads; i++) {\n \t\tstruct thread_data *p = data+i;\n \t\tp->index = index;\n@@ -100,6 +135,8 @@ static void preload_index(struct index_state *index,\n \t\t\tcopy_pathspec(&p->pathspec, pathspec);\n \t\tp->offset = offset;\n \t\tp->nr = work;\n+\t\tif (pd.progress)\n+\t\t\tp->progress = &pd;\n \t\toffset += work;\n \t\tif (pthread_create(&p->pthread, NULL, preload_thread, p))\n \t\t\tdie(\"unable to create threaded lstat\");\n@@ -109,15 +146,18 @@ static void preload_index(struct index_state *index,\n \t\tif (pthread_join(p->pthread, NULL))\n \t\t\tdie(\"unable to join threaded lstat\");\n \t}\n+\tstop_progress(&pd.progress);\n+\n \ttrace_performance_since(start, \"preload index\");\n }\n #endif\n \n int read_index_preload(struct index_state *index,\n-\t\t       const struct pathspec *pathspec)\n+\t\t       const struct pathspec *pathspec,\n+\t\t       unsigned int refresh_flags)\n {\n \tint retval = read_index(index);\n \n-\tpreload_index(index, pathspec);\n+\tpreload_index(index, pathspec, refresh_flags);\n \treturn retval;\n }\ndiff --git a/read-cache.c b/read-cache.c\nindex 7b1354d759..5969ca93c7 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -23,6 +23,7 @@\n #include \"split-index.h\"\n #include \"utf8.h\"\n #include \"fsmonitor.h\"\n+#include \"progress.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -1477,6 +1478,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \tconst char *added_fmt;\n \tconst char *unmerged_fmt;\n \tuint64_t start = getnanotime();\n+\tstruct progress *progress = NULL;\n+\n+\tif (flags & REFRESH_PROGRESS && isatty(2))\n+\t\tprogress = start_delayed_progress(_(\"Refresh index\"),\n+\t\t\t\t\t\t  istate->cache_nr);\n \n \tmodified_fmt = (in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\");\n \tdeleted_fmt = (in_porcelain ? \"D\\t%s\\n\" : \"%s: needs update\\n\");\n@@ -1516,6 +1522,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tnew_entry = refresh_cache_ent(istate, ce, options, &cache_errno, &changed);\n \t\tif (new_entry == ce)\n \t\t\tcontinue;\n+\t\tif (progress)\n+\t\t\tdisplay_progress(progress, i);\n \t\tif (!new_entry) {\n \t\t\tconst char *fmt;\n \n@@ -1547,6 +1555,10 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \n \t\treplace_index_entry(istate, i, new_entry);\n \t}\n+\tif (progress) {\n+\t\tdisplay_progress(progress, istate->cache_nr);\n+\t\tstop_progress(&progress);\n+\t}\n \ttrace_performance_since(start, \"refresh index\");\n \treturn has_errors;\n }\ndiff --git a/sequencer.c b/sequencer.c\nindex dc2c58d464..e0cd17df70 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -1909,7 +1909,7 @@ static int read_and_refresh_cache(struct replay_opts *opts)\n {\n \tstruct lock_file index_lock = LOCK_INIT;\n \tint index_fd = hold_locked_index(&index_lock, 0);\n-\tif (read_index_preload(&the_index, NULL) < 0) {\n+\tif (read_index_preload(&the_index, NULL, 0) < 0) {\n \t\trollback_lock_file(&index_lock);\n \t\treturn error(_(\"git %s: failed to read the index\"),\n \t\t\t_(action_name(opts)));\n-- \n2.19.0.rc0.337.ge906d732e7\n\n"}]}