{"thread":{"id":"49711","subject":"[PATCH v1] speed up refresh_index() by utilizing preload_index()","startedAt":"2018-10-29T20:42:26Z","lastAt":"2018-10-30T02:28:33Z","messageCount":2,"participants":["Ben Peart","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"361942","messageId":"20181029204159.18208-1-peartben@gmail.com","threadId":"49711","inReplyTo":null,"subject":"[PATCH v1] speed up refresh_index() by utilizing preload_index()","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-10-29T20:41:59Z","receivedAt":"2018-10-29T20:42:26Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"From: Ben Peart <benpeart@microsoft.com>\n\nSpeed up refresh_index() by utilizing preload_index() to do most of the work\nspread across multiple threads.  This works because most cache entries will\nget marked CE_UPTODATE so that refresh_cache_ent() can bail out early when\ncalled from within refresh_index().\n\nOn a Windows repo with ~200K files, this drops refresh times from 6.64\nseconds to 2.87 seconds for a savings of 57%.\n\nSigned-off-by: Ben Peart <benpeart@microsoft.com>\n---\n\nNotes:\n    Base Ref: master\n    Web-Diff: https://github.com/benpeart/git/commit/feee1054c2\n    Checkout: git fetch https://github.com/benpeart/git refresh-index-multithread-preload-v1 && git checkout feee1054c2\n\n cache.h         | 3 +++\n preload-index.c | 8 ++++----\n read-cache.c    | 6 ++++++\n 3 files changed, 13 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex f7fabdde8f..883099db08 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -659,6 +659,9 @@ extern int daemonize(void);\n /* Initialize and use the cache information */\n struct lock_file;\n extern int read_index(struct index_state *);\n+extern void preload_index(struct index_state *index,\n+\t\t\t  const struct pathspec *pathspec,\n+\t\t\t  unsigned int refresh_flags);\n extern int read_index_preload(struct index_state *,\n \t\t\t      const struct pathspec *pathspec,\n \t\t\t      unsigned int refresh_flags);\ndiff --git a/preload-index.c b/preload-index.c\nindex 9e7152ab14..222792ccbc 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -9,7 +9,7 @@\n #include \"progress.h\"\n \n #ifdef NO_PTHREADS\n-static void preload_index(struct index_state *index,\n+void preload_index(struct index_state *index,\n \t\t\t  const struct pathspec *pathspec,\n \t\t\t  unsigned int refresh_flags)\n {\n@@ -100,9 +100,9 @@ static void *preload_thread(void *_data)\n \treturn NULL;\n }\n \n-static void preload_index(struct index_state *index,\n-\t\t\t  const struct pathspec *pathspec,\n-\t\t\t  unsigned int refresh_flags)\n+void preload_index(struct index_state *index,\n+\t\t   const struct pathspec *pathspec,\n+\t\t   unsigned int refresh_flags)\n {\n \tint threads, i, work, offset;\n \tstruct thread_data data[MAX_PARALLEL];\ndiff --git a/read-cache.c b/read-cache.c\nindex d57958233e..53733d651d 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1496,6 +1496,12 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \ttypechange_fmt = (in_porcelain ? \"T\\t%s\\n\" : \"%s needs update\\n\");\n \tadded_fmt = (in_porcelain ? \"A\\t%s\\n\" : \"%s needs update\\n\");\n \tunmerged_fmt = (in_porcelain ? \"U\\t%s\\n\" : \"%s: needs merge\\n\");\n+\t/*\n+\t * Use the multi-threaded preload_index() to refresh most of the\n+\t * cache entries quickly then in the single threaded loop below,\n+\t * we only have to do the special cases that are left.\n+\t */\n+\tpreload_index(istate, pathspec, 0);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n\nbase-commit: c670b1f876521c9f7cd40184bf7ed05aad843433\n-- \n2.9.2.gvfs.2.27918.g0990287eef\n\n"},{"id":"361958","messageId":"xmqq7ei02mnn.fsf@gitster-ct.c.googlers.com","threadId":"49711","inReplyTo":"20181029204159.18208-1-peartben@gmail.com","subject":"Re: [PATCH v1] speed up refresh_index() by utilizing preload_index()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-30T02:28:28Z","receivedAt":"2018-10-30T02:28:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart <peartben@gmail.com> writes:\n\n> From: Ben Peart <benpeart@microsoft.com>\n>\n> Speed up refresh_index() by utilizing preload_index() to do most of the work\n> spread across multiple threads.  This works because most cache entries will\n> get marked CE_UPTODATE so that refresh_cache_ent() can bail out early when\n> called from within refresh_index().\n>\n> On a Windows repo with ~200K files, this drops refresh times from 6.64\n> seconds to 2.87 seconds for a savings of 57%.\n>\n> Signed-off-by: Ben Peart <benpeart@microsoft.com>\n> ---\n\nOK.  We used to only expose the whole \"read the index file into an\nistate, and then do the lstat() part in parallel\", but now we also\nmake the \"do the lstat() part\" available separately.\n\nWhich makes sense.\n\n\n> diff --git a/cache.h b/cache.h\n> index f7fabdde8f..883099db08 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -659,6 +659,9 @@ extern int daemonize(void);\n>  /* Initialize and use the cache information */\n>  struct lock_file;\n>  extern int read_index(struct index_state *);\n> +extern void preload_index(struct index_state *index,\n> +\t\t\t  const struct pathspec *pathspec,\n> +\t\t\t  unsigned int refresh_flags);\n>  extern int read_index_preload(struct index_state *,\n>  \t\t\t      const struct pathspec *pathspec,\n>  \t\t\t      unsigned int refresh_flags);\n> diff --git a/preload-index.c b/preload-index.c\n> index 9e7152ab14..222792ccbc 100644\n> --- a/preload-index.c\n> +++ b/preload-index.c\n> @@ -9,7 +9,7 @@\n>  #include \"progress.h\"\n>  \n>  #ifdef NO_PTHREADS\n> -static void preload_index(struct index_state *index,\n> +void preload_index(struct index_state *index,\n>  \t\t\t  const struct pathspec *pathspec,\n>  \t\t\t  unsigned int refresh_flags)\n>  {\n> @@ -100,9 +100,9 @@ static void *preload_thread(void *_data)\n>  \treturn NULL;\n>  }\n>  \n> -static void preload_index(struct index_state *index,\n> -\t\t\t  const struct pathspec *pathspec,\n> -\t\t\t  unsigned int refresh_flags)\n> +void preload_index(struct index_state *index,\n> +\t\t   const struct pathspec *pathspec,\n> +\t\t   unsigned int refresh_flags)\n>  {\n>  \tint threads, i, work, offset;\n>  \tstruct thread_data data[MAX_PARALLEL];\n> diff --git a/read-cache.c b/read-cache.c\n> index d57958233e..53733d651d 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1496,6 +1496,12 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n>  \ttypechange_fmt = (in_porcelain ? \"T\\t%s\\n\" : \"%s needs update\\n\");\n>  \tadded_fmt = (in_porcelain ? \"A\\t%s\\n\" : \"%s needs update\\n\");\n>  \tunmerged_fmt = (in_porcelain ? \"U\\t%s\\n\" : \"%s: needs merge\\n\");\n> +\t/*\n> +\t * Use the multi-threaded preload_index() to refresh most of the\n> +\t * cache entries quickly then in the single threaded loop below,\n> +\t * we only have to do the special cases that are left.\n> +\t */\n> +\tpreload_index(istate, pathspec, 0);\n>  \tfor (i = 0; i < istate->cache_nr; i++) {\n>  \t\tstruct cache_entry *ce, *new_entry;\n>  \t\tint cache_errno = 0;\n>\n> base-commit: c670b1f876521c9f7cd40184bf7ed05aad843433\n"}]}