{"thread":{"id":"28646","subject":"[PATCH] Improving performance with pthreads in refresh_index().","startedAt":"2011-10-11T09:32:01Z","lastAt":"2011-10-12T07:21:20Z","messageCount":5,"participants":["klinkert@webgods.de","Johannes Sixt","Nguyen Thai Ngoc Duy","Michael J Gruber","Simon Klinkert"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"177362","messageId":"1318325521-23262-1-git-send-email-klinkert@webgods.de","threadId":"28646","inReplyTo":null,"subject":"[PATCH] Improving performance with pthreads in refresh_index().","fromName":"","fromEmail":"klinkert@webgods.de","sentAt":"2011-10-11T09:32:01Z","receivedAt":"2011-10-11T09:32:01Z","isPatch":true,"sender":{"key":"klinkert@webgods.de","avatar":null},"body":"Git performs for every file in a repository at least one (with a cold cache)\nlstat(). In larger repositories operations like git status take a\nlong time. In case your local repository is located on a remote server\n(e. g. mounted via nfs) it ends up in an *incredible* slow git.\n\nWith this patch you're able to determine a number of threads (maxthreads)\nin your config file to run these tons of lstats in threads. There\nwon't be created any pthreads if you haven't set maxthreads. In my\ntest cases a git status with this patch performs enormously faster (over\ntwo minutes before and approximately 25 seconds now). Of course, it\nhas a positive impact on other git commands, too.\n\nSigned-off-by: Simon Klinkert <klinkert@webgods.de>\n---\n attr.c        |   22 +++++++++\n cache.h       |    1 +\n config.c      |    5 ++\n environment.c |    1 +\n read-cache.c  |  140 +++++++++++++++++++++++++++++++++++++++++++++++++-------\n 5 files changed, 151 insertions(+), 18 deletions(-)\n\ndiff --git a/attr.c b/attr.c\nindex 33cb4e4..d296fe8 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -8,10 +8,14 @@\n  */\n \n #define NO_THE_INDEX_COMPATIBILITY_MACROS\n+#include <pthread.h>\n+#undef _FILE_OFFSET_BITS\n #include \"cache.h\"\n #include \"exec_cmd.h\"\n #include \"attr.h\"\n \n+pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;\n+\n const char git_attr__true[] = \"(builtin)true\";\n const char git_attr__false[] = \"\\0(builtin)false\";\n static const char git_attr__unknown[] = \"(builtin)unknown\";\n@@ -748,6 +752,9 @@ int git_check_attr(const char *path, int num, struct git_attr_check *check)\n {\n \tint i;\n \n+\tif (max_threads)\n+\t\tpthread_mutex_lock(&mutex);\n+\n \tcollect_all_attrs(path);\n \n \tfor (i = 0; i < num; i++) {\n@@ -757,6 +764,9 @@ int git_check_attr(const char *path, int num, struct git_attr_check *check)\n \t\tcheck[i].value = value;\n \t}\n \n+\tif (max_threads)\n+\t\tpthread_mutex_unlock(&mutex);\n+\n \treturn 0;\n }\n \n@@ -764,6 +774,9 @@ int git_all_attrs(const char *path, int *num, struct git_attr_check **check)\n {\n \tint i, count, j;\n \n+\tif (max_threads)\n+\t\tpthread_mutex_lock(&mutex);\n+\n \tcollect_all_attrs(path);\n \n \t/* Count the number of attributes that are set. */\n@@ -785,6 +798,9 @@ int git_all_attrs(const char *path, int *num, struct git_attr_check **check)\n \t\t}\n \t}\n \n+\tif (max_threads)\n+\t\tpthread_mutex_unlock(&mutex);\n+\n \treturn 0;\n }\n \n@@ -795,8 +811,14 @@ void git_attr_set_direction(enum git_attr_direction new, struct index_state *ist\n \tif (is_bare_repository() && new != GIT_ATTR_INDEX)\n \t\tdie(\"BUG: non-INDEX attr direction in a bare repo\");\n \n+\tif (max_threads)\n+\t\tpthread_mutex_lock(&mutex);\n+\n \tdirection = new;\n \tif (new != old)\n \t\tdrop_attr_stack();\n \tuse_index = istate;\n+\n+\tif (max_threads)\n+\t\tpthread_mutex_unlock(&mutex);\n }\ndiff --git a/cache.h b/cache.h\nindex 607c2ea..ab1b3e4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -600,6 +600,7 @@ extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\n+extern int max_threads;\n \n enum branch_track {\n \tBRANCH_TRACK_UNSPECIFIED = -1,\ndiff --git a/config.c b/config.c\nindex 4183f80..24de139 100644\n--- a/config.c\n+++ b/config.c\n@@ -466,6 +466,11 @@ int git_config_pathname(const char **dest, const char *var, const char *value)\n \n static int git_default_core_config(const char *var, const char *value)\n {\n+\tif (!strcmp(var, \"core.maxthreads\")) {\n+\t\tmax_threads = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n \t/* This needs a better name */\n \tif (!strcmp(var, \"core.filemode\")) {\n \t\ttrust_executable_bit = git_config_bool(var, value);\ndiff --git a/environment.c b/environment.c\nindex e96edcf..8bca4d5 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -59,6 +59,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n struct startup_info *startup_info;\n+int max_threads = 1;\n \n /* Parallel index stat data preload? */\n int core_preload_index = 0;\ndiff --git a/read-cache.c b/read-cache.c\nindex 01a0e25..350bf4b 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -5,6 +5,8 @@\n  */\n #define NO_THE_INDEX_COMPATIBILITY_MACROS\n #include \"cache.h\"\n+#undef _FILE_OFFSET_BITS\n+#include <pthread.h>\n #include \"cache-tree.h\"\n #include \"refs.h\"\n #include \"dir.h\"\n@@ -13,6 +15,8 @@\n #include \"blob.h\"\n #include \"resolve-undo.h\"\n \n+pthread_mutex_t mutex_refresh_index = PTHREAD_MUTEX_INITIALIZER;\n+\n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int really);\n \n /* Index extensions.\n@@ -1080,7 +1084,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n }\n \n static void show_file(const char * fmt, const char * name, int in_porcelain,\n-\t\t      int * first, const char *header_msg)\n+\t\t      int * first, char *header_msg)\n {\n \tif (in_porcelain && *first && header_msg) {\n \t\tprintf(\"%s\\n\", header_msg);\n@@ -1089,45 +1093,96 @@ static void show_file(const char * fmt, const char * name, int in_porcelain,\n \tprintf(fmt, name);\n }\n \n-int refresh_index(struct index_state *istate, unsigned int flags, const char **pathspec,\n-\t\t  char *seen, const char *header_msg)\n+struct t_ctx {\n+\tint flags;\n+\tint has_errors;\n+\tstruct index_state *istate;\n+\tconst char **pathspec;\n+\tchar *seen, header_msg;\n+\n+\tint tasks_done;\n+} ctx;\n+\n+int\n+thread_manager(void)\n+{\n+\tif (max_threads)\n+\t\tpthread_mutex_lock(&mutex_refresh_index);\n+\n+\tif (ctx.tasks_done >= ctx.istate->cache_nr) {\n+\t\tif (max_threads)\n+\t\t\tpthread_mutex_unlock(&mutex_refresh_index);\n+\t\treturn -1;\n+\t}\n+\n+\tint task = ctx.tasks_done++;\n+\n+\tif (max_threads)\n+\t\tpthread_mutex_unlock(&mutex_refresh_index);\n+\n+\treturn task;\n+}\n+\n+void *\n+thread_refresh_index(void *p)\n {\n+\tstruct cache_entry *ce, *new;\n+\tint cache_errno = 0;\n \tint i;\n-\tint has_errors = 0;\n+\n+\twhile ((i = thread_manager()) != -1) {\n+\t\tstruct index_state *istate = ctx.istate;\n+\t\tint flags = ctx.flags;\n+\n \tint really = (flags & REFRESH_REALLY) != 0;\n \tint allow_unmerged = (flags & REFRESH_UNMERGED) != 0;\n \tint quiet = (flags & REFRESH_QUIET) != 0;\n \tint not_new = (flags & REFRESH_IGNORE_MISSING) != 0;\n-\tint ignore_submodules = (flags & REFRESH_IGNORE_SUBMODULES) != 0;\n+\t\tint ignore_submodules =\n+\t\t\t (flags & REFRESH_IGNORE_SUBMODULES) != 0;\n \tint first = 1;\n \tint in_porcelain = (flags & REFRESH_IN_PORCELAIN);\n \tunsigned int options = really ? CE_MATCH_IGNORE_VALID : 0;\n \tconst char *needs_update_fmt;\n \tconst char *needs_merge_fmt;\n \n-\tneeds_update_fmt = (in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\");\n-\tneeds_merge_fmt = (in_porcelain ? \"U\\t%s\\n\" : \"%s: needs merge\\n\");\n-\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\tstruct cache_entry *ce, *new;\n-\t\tint cache_errno = 0;\n+\t\tneeds_update_fmt =\n+\t\t\t (in_porcelain ? \"M\\t%s\\n\" : \"%s: needs update\\n\");\n+\t\tneeds_merge_fmt =\n+\t\t\t (in_porcelain ? \"U\\t%s\\n\" : \"%s: needs merge\\n\");\n \n \t\tce = istate->cache[i];\n-\t\tif (ignore_submodules && S_ISGITLINK(ce->ce_mode))\n+\t\tif (ignore_submodules && S_ISGITLINK(ce->ce_mode)) {\n \t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (max_threads)\n+\t\t\tpthread_mutex_lock(&mutex_refresh_index);\n \n \t\tif (ce_stage(ce)) {\n \t\t\twhile ((i < istate->cache_nr) &&\n \t\t\t       ! strcmp(istate->cache[i]->name, ce->name))\n \t\t\t\ti++;\n \t\t\ti--;\n-\t\t\tif (allow_unmerged)\n+\t\t\tif (allow_unmerged) {\n+\t\t\t\tif (max_threads)\n+\t\t\t\t\tpthread_mutex_unlock(&mutex_refresh_index);\n \t\t\t\tcontinue;\n-\t\t\tshow_file(needs_merge_fmt, ce->name, in_porcelain, &first, header_msg);\n-\t\t\thas_errors = 1;\n+\t\t\t}\n+\t\t\tshow_file(needs_merge_fmt, ce->name, in_porcelain,\n+\t\t\t\t &first, &ctx.header_msg);\n+\t\t\tctx.has_errors = 1;\n+\t\t\tif (max_threads)\n+\t\t\t\tpthread_mutex_unlock(&mutex_refresh_index);\n \t\t\tcontinue;\n \t\t}\n \n-\t\tif (pathspec && !match_pathspec(pathspec, ce->name, strlen(ce->name), 0, seen))\n+\t\tif (max_threads)\n+\t\t\tpthread_mutex_unlock(&mutex_refresh_index);\n+\n+\t\tif (ctx.pathspec && !match_pathspec(ctx.pathspec, ce->name,\n+\t\t\t\t\t\t    strlen(ce->name), 0,\n+\t\t\t\t\t\t    ctx.seen))\n \t\t\tcontinue;\n \n \t\tnew = refresh_cache_ent(istate, ce, options, &cache_errno);\n@@ -1145,14 +1200,63 @@ int refresh_index(struct index_state *istate, unsigned int flags, const char **p\n \t\t\t}\n \t\t\tif (quiet)\n \t\t\t\tcontinue;\n-\t\t\tshow_file(needs_update_fmt, ce->name, in_porcelain, &first, header_msg);\n-\t\t\thas_errors = 1;\n+\n+\t\t\tif (max_threads)\n+\t\t\t\tpthread_mutex_lock(&mutex_refresh_index);\n+\n+\t\t\tshow_file(needs_update_fmt, ce->name, in_porcelain,\n+\t\t\t\t  &first, &ctx.header_msg);\n+\n+\t\t\tif (max_threads)\n+\t\t\t\tpthread_mutex_unlock(&mutex_refresh_index);\n+\n+\t\t\tctx.has_errors = 1;\n \t\t\tcontinue;\n \t\t}\n \n \t\treplace_index_entry(istate, i, new);\n \t}\n-\treturn has_errors;\n+\treturn NULL;\n+}\n+\n+int refresh_index(struct index_state *istate, unsigned int flags,\n+\t\t  const char **pathspec, char *seen, const char *header_msg)\n+{\n+\tint i;\n+\tint ret;\n+\tunsigned int created_threads = 0;\n+\n+\tctx.has_errors = 0;\n+\tctx.tasks_done = 0;\n+\tctx.istate = istate;\n+\tctx.flags = flags;\n+\tctx.pathspec = &pathspec[0];\n+\n+\tif (istate->cache_nr < max_threads)\n+\t\tmax_threads = istate->cache_nr;\n+\n+\tif (max_threads > 1) {\n+\t\tpthread_t threads[max_threads];\n+\n+\t\t/* create threads */\n+\t\tfor (i = 0; i < max_threads; i++) {\n+\t\t\tret = pthread_create(&threads[created_threads], NULL,\n+\t\t\t\t\t     thread_refresh_index, NULL);\n+\t\t\tif (ret) {\n+\t\t\t\tprintf(\"pthread_create failed ret=%d\\n\", ret);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\t++created_threads;\n+\t\t}\n+\n+\t\t/* collect threads */\n+\t\tfor (i = 0; i < created_threads; i++) {\n+\t\t\tret = pthread_join(threads[i], NULL);\n+\t\t}\n+\t} else {\n+\t\tthread_refresh_index(NULL);\n+\t}\n+\treturn ctx.has_errors;\n }\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int really)\n-- \n1.7.7\n"},{"id":"177363","messageId":"4E941E97.2070906@viscovery.net","threadId":"28646","inReplyTo":"1318325521-23262-1-git-send-email-klinkert@webgods.de","subject":"Re: [PATCH] Improving performance with pthreads in refresh_index().","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2011-10-11T10:46:47Z","receivedAt":"2011-10-11T10:46:47Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"First of all, thanks for your contribution!\n\nWhen you submit a patch, you should make sure that the \"From:\" line of\nyour mail includes your full name. You included it in the Signed-off-by\nline and that is a good start.\n\nAm 10/11/2011 11:32, schrieb klinkert@webgods.de:\n> Git performs for every file in a repository at least one (with a cold cache)\n> lstat().\n\nIt doesn't do the lstat() when the cache is warm? I doubt it.\n\n> In larger repositories operations like git status take a\n> long time. In case your local repository is located on a remote server\n> (e. g. mounted via nfs) it ends up in an *incredible* slow git.\n\n\"Incredible\" is very subjective. You should back your claim with benchmarks.\n\n> With this patch you're able to determine a number of threads (maxthreads)\n> in your config file to run these tons of lstats in threads. There\n> won't be created any pthreads if you haven't set maxthreads. In my\n> test cases a git status with this patch performs enormously faster (over\n> two minutes before and approximately 25 seconds now).\n\nOk, so here you have something that you can turn into a benchmark. Just\nreplace the hand-waving by hard facts.\n\nYou report an improvement by a factor 4. How many threads did you use? How\ndoes the number of threads change the improvement. How does the number of\nthreads influence the performance on a repository that is not on a network\npartition? Can I set it to 25 even on a dual core machine without\nnoticable degradation?\n\n> Of course, it\n> has a positive impact on other git commands, too.\n\nWhat other commands? Benchmarks?\n\n> diff --git a/attr.c b/attr.c\n> index 33cb4e4..d296fe8 100644\n> --- a/attr.c\n> +++ b/attr.c\n> @@ -8,10 +8,14 @@\n>   */\n>  \n>  #define NO_THE_INDEX_COMPATIBILITY_MACROS\n> +#include <pthread.h>\n\nThis and all pthread usages should be bracketed by NO_PTHREADS, or you\ncould use thread-utils.h instead. And it should be included *after* cache.h.\n\n> +#undef _FILE_OFFSET_BITS\n\nWhat is this good for?\n\n>  #include \"cache.h\"\n>  #include \"exec_cmd.h\"\n>  #include \"attr.h\"\n>  \n> +pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;\n\nThis should be static, no? Otherwise, the name is *WAY* too generic for a\nglobal variable.\n\nWhat does the mutex protect? Please write some comment. It is not obvious.\nOr perhaps it is sufficient to move the variable near the data that it\nprotects.\n\nPTHREAD_MUTEX_INITIALIZER does not work on Windows. Please call\npthread_mutex_init() from a suitable place.\n\n> +\n>  const char git_attr__true[] = \"(builtin)true\";\n>  const char git_attr__false[] = \"\\0(builtin)false\";\n>  static const char git_attr__unknown[] = \"(builtin)unknown\";\n> @@ -748,6 +752,9 @@ int git_check_attr(const char *path, int num, struct git_attr_check *check)\n>  {\n>  \tint i;\n>  \n> +\tif (max_threads)\n> +\t\tpthread_mutex_lock(&mutex);\n\nAs I said, this should be bracketed by #ifndef NO_PTHREADS. But since you\nalso bracket all mutex uses by max_threads anyway, you should factor this\ninto a small helper function.\n\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -600,6 +600,7 @@ extern int read_replace_refs;\n>  extern int fsync_object_files;\n>  extern int core_preload_index;\n>  extern int core_apply_sparse_checkout;\n> +extern int max_threads;\n\nThis name is *WAY* too generic for a global variable. Please choose a more\nspecific name.\n\n> --- a/config.c\n> +++ b/config.c\n> @@ -466,6 +466,11 @@ int git_config_pathname(const char **dest, const char *var, const char *value)\n>  \n>  static int git_default_core_config(const char *var, const char *value)\n>  {\n> +\tif (!strcmp(var, \"core.maxthreads\")) {\n\nSame here. core.maxthreads is *WAY* too generic for its current use.\nPlease paint the bikeshed in a more distiguishing color. ;)\n\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -5,6 +5,8 @@\n>   */\n>  #define NO_THE_INDEX_COMPATIBILITY_MACROS\n>  #include \"cache.h\"\n> +#undef _FILE_OFFSET_BITS\n\nAgain: why this?\n\n> +#include <pthread.h>\n>  #include \"cache-tree.h\"\n>  #include \"refs.h\"\n>  #include \"dir.h\"\n> @@ -13,6 +15,8 @@\n>  #include \"blob.h\"\n>  #include \"resolve-undo.h\"\n>  \n> +pthread_mutex_t mutex_refresh_index = PTHREAD_MUTEX_INITIALIZER;\n\nShould be static, no?\n\n>  static void show_file(const char * fmt, const char * name, int in_porcelain,\n> -\t\t      int * first, const char *header_msg)\n> +\t\t      int * first, char *header_msg)\n\nWhy?\n\n> -int refresh_index(struct index_state *istate, unsigned int flags, const char **pathspec,\n> -\t\t  char *seen, const char *header_msg)\n> +struct t_ctx {\n> +\tint flags;\n> +\tint has_errors;\n> +\tstruct index_state *istate;\n> +\tconst char **pathspec;\n> +\tchar *seen, header_msg;\n> +\n> +\tint tasks_done;\n> +} ctx;\n\nShould this be static? Is this the only global data that is protected by\nthe mutex?\n\n> +\n> +int\n> +thread_manager(void)\n\nStyle: the type is on the same line as the function name.\n\n> +void *\n> +thread_refresh_index(void *p)\n>  {\n> +\tstruct cache_entry *ce, *new;\n> +\tint cache_errno = 0;\n>  \tint i;\n> -\tint has_errors = 0;\n> +\n> +\twhile ((i = thread_manager()) != -1) {\n> +\t\tstruct index_state *istate = ctx.istate;\n> +\t\tint flags = ctx.flags;\n> +\n>  \tint really = (flags & REFRESH_REALLY) != 0;\n>  \tint allow_unmerged = (flags & REFRESH_UNMERGED) != 0;\n>  \tint quiet = (flags & REFRESH_QUIET) != 0;\n>  \tint not_new = (flags & REFRESH_IGNORE_MISSING) != 0;\n> -\tint ignore_submodules = (flags & REFRESH_IGNORE_SUBMODULES) != 0;\n> +\t\tint ignore_submodules =\n> +\t\t\t (flags & REFRESH_IGNORE_SUBMODULES) != 0;\n\nIndentation?\n\nPerhaps it is worthwhile to factor the body of the loop into a helper\nfunction in a preparatory patch to reduce churn in the \"real\" patch.\n\n>  \t\tce = istate->cache[i];\n> -\t\tif (ignore_submodules && S_ISGITLINK(ce->ce_mode))\n> +\t\tif (ignore_submodules && S_ISGITLINK(ce->ce_mode)) {\n>  \t\t\tcontinue;\n> +\t\t}\n\nStyle: do not add braces unnecessarily.\n\n> +\n> +\t\tif (max_threads)\n> +\t\t\tpthread_mutex_lock(&mutex_refresh_index);\n>  \n>  \t\tif (ce_stage(ce)) {\n>  \t\t\twhile ((i < istate->cache_nr) &&\n>  \t\t\t       ! strcmp(istate->cache[i]->name, ce->name))\n>  \t\t\t\ti++;\n>  \t\t\ti--;\n> -\t\t\tif (allow_unmerged)\n> +\t\t\tif (allow_unmerged) {\n> +\t\t\t\tif (max_threads)\n> +\t\t\t\t\tpthread_mutex_unlock(&mutex_refresh_index);\n>  \t\t\t\tcontinue;\n> -\t\t\tshow_file(needs_merge_fmt, ce->name, in_porcelain, &first, header_msg);\n> -\t\t\thas_errors = 1;\n> +\t\t\t}\n> +\t\t\tshow_file(needs_merge_fmt, ce->name, in_porcelain,\n> +\t\t\t\t &first, &ctx.header_msg);\n\nYou are calling show_file() from the thread. Does that mean that the\noutput is not in index order anymore? Are there guarantees that the output\nof different threads does not overlap?\n\n> +\t\t\tctx.has_errors = 1;\n> +\t\t\tif (max_threads)\n> +\t\t\t\tpthread_mutex_unlock(&mutex_refresh_index);\n\nSo, at least the second worry is unfounded since output is produced while\nthe mutex is held.\n\nBTW, would it be possible to use a different mutex for this purpose?\n\n> +\t\t/* create threads */\n> +\t\tfor (i = 0; i < max_threads; i++) {\n> +\t\t\tret = pthread_create(&threads[created_threads], NULL,\n> +\t\t\t\t\t     thread_refresh_index, NULL);\n> +\t\t\tif (ret) {\n> +\t\t\t\tprintf(\"pthread_create failed ret=%d\\n\", ret);\n\nWe have warning() to write warnings.\n\n-- Hannes\n"},{"id":"177364","messageId":"CACsJy8D7-vtfNvx8BqAK77Y7A4qYxORTzovMugXRZLuxXtYY+Q@mail.gmail.com","threadId":"28646","inReplyTo":"1318325521-23262-1-git-send-email-klinkert@webgods.de","subject":"Re: [PATCH] Improving performance with pthreads in refresh_index().","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-10-11T10:59:01Z","receivedAt":"2011-10-11T10:59:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Oct 11, 2011 at 8:32 PM,  <klinkert@webgods.de> wrote:\n> Git performs for every file in a repository at least one (with a cold cache)\n> lstat(). In larger repositories operations like git status take a\n> long time. In case your local repository is located on a remote server\n> (e. g. mounted via nfs) it ends up in an *incredible* slow git.\n\nThis sounds really similar to what Linus did with preload-index.c..\n-- \nDuy\n"},{"id":"177365","messageId":"4E942D76.7030908@drmicha.warpmail.net","threadId":"28646","inReplyTo":"1318325521-23262-1-git-send-email-klinkert@webgods.de","subject":"Re: [PATCH] Improving performance with pthreads in refresh_index().","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2011-10-11T11:50:14Z","receivedAt":"2011-10-11T11:50:14Z","isPatch":true,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"klinkert@webgods.de venit, vidit, dixit 11.10.2011 11:32:\n> Git performs for every file in a repository at least one (with a cold cache)\n> lstat(). In larger repositories operations like git status take a\n> long time. In case your local repository is located on a remote server\n> (e. g. mounted via nfs) it ends up in an *incredible* slow git.\n> \n> With this patch you're able to determine a number of threads (maxthreads)\n> in your config file to run these tons of lstats in threads. There\n> won't be created any pthreads if you haven't set maxthreads. In my\n> test cases a git status with this patch performs enormously faster (over\n> two minutes before and approximately 25 seconds now). Of course, it\n> has a positive impact on other git commands, too.\n\nCan you specify under which circumstances one should get a speedup? Our\nNFS isn't slow enough... but on a dead slow sshfs work tree I get the\nfollowing for \"git status -s\":\n\nmaxthreads: 0, preloadindex: false, time: 14.73\nmaxthreads: 1, preloadindex: false, time: 14.25\nmaxthreads: 2, preloadindex: false, time: 13.32\nmaxthreads: 3, preloadindex: false, time: 12.40\nmaxthreads: 4, preloadindex: false, time: 12.65\nmaxthreads: 5, preloadindex: false, time: 12.16\nmaxthreads: 8, preloadindex: false, time: 12.32\nmaxthreads: 10, preloadindex: false, time: 11.98\nmaxthreads: 15, preloadindex: false, time: 12.31\nmaxthreads: 20, preloadindex: false, time: 12.00\nmaxthreads: 0, preloadindex: true, time: 12.17\nmaxthreads: 1, preloadindex: true, time: 11.98\nmaxthreads: 2, preloadindex: true, time: 12.21\nmaxthreads: 3, preloadindex: true, time: 11.99\nmaxthreads: 4, preloadindex: true, time: 12.14\nmaxthreads: 5, preloadindex: true, time: 12.21\nmaxthreads: 8, preloadindex: true, time: 12.14\nmaxthreads: 10, preloadindex: true, time: 12.08\nmaxthreads: 15, preloadindex: true, time: 12.16\nmaxthreads: 20, preloadindex: true, time: 11.96\n\nSo it seams it gives me what preloadindex does, which is not much.\n\nNote: I'm not saying the patch is bad. I'm just wondering whether that\nis expected.\n\nMichael\nP.S.: It's actually sshfs with ssh to an NFSv3 client (server restricts\nexports) :(\n"},{"id":"177428","messageId":"470EDA14-707D-42C7-A57A-CFE7208E1D03@webgods.de","threadId":"28646","inReplyTo":"4E942D76.7030908@drmicha.warpmail.net","subject":"Re: [PATCH] Improving performance with pthreads in refresh_index().","fromName":"Simon Klinkert","fromEmail":"klinkert@webgods.de","sentAt":"2011-10-12T07:21:20Z","receivedAt":"2011-10-12T07:21:20Z","isPatch":true,"sender":{"key":"klinkert@webgods.de","avatar":null},"body":"\nOkay. It seems like my idea was already implemented by Linus. I wasn't aware of that fact.\n\nAnyway, I've learned a bit more about git's bowels. Thanks a lot.\n\nSimon\n\nOn 11.10.2011, at 13:50, Michael J Gruber wrote:\n\n> klinkert@webgods.de venit, vidit, dixit 11.10.2011 11:32:\n>> Git performs for every file in a repository at least one (with a cold cache)\n>> lstat(). In larger repositories operations like git status take a\n>> long time. In case your local repository is located on a remote server\n>> (e. g. mounted via nfs) it ends up in an *incredible* slow git.\n>> \n>> With this patch you're able to determine a number of threads (maxthreads)\n>> in your config file to run these tons of lstats in threads. There\n>> won't be created any pthreads if you haven't set maxthreads. In my\n>> test cases a git status with this patch performs enormously faster (over\n>> two minutes before and approximately 25 seconds now). Of course, it\n>> has a positive impact on other git commands, too.\n> \n> Can you specify under which circumstances one should get a speedup? Our\n> NFS isn't slow enough... but on a dead slow sshfs work tree I get the\n> following for \"git status -s\":\n> \n> maxthreads: 0, preloadindex: false, time: 14.73\n> maxthreads: 1, preloadindex: false, time: 14.25\n> maxthreads: 2, preloadindex: false, time: 13.32\n> maxthreads: 3, preloadindex: false, time: 12.40\n> maxthreads: 4, preloadindex: false, time: 12.65\n> maxthreads: 5, preloadindex: false, time: 12.16\n> maxthreads: 8, preloadindex: false, time: 12.32\n> maxthreads: 10, preloadindex: false, time: 11.98\n> maxthreads: 15, preloadindex: false, time: 12.31\n> maxthreads: 20, preloadindex: false, time: 12.00\n> maxthreads: 0, preloadindex: true, time: 12.17\n> maxthreads: 1, preloadindex: true, time: 11.98\n> maxthreads: 2, preloadindex: true, time: 12.21\n> maxthreads: 3, preloadindex: true, time: 11.99\n> maxthreads: 4, preloadindex: true, time: 12.14\n> maxthreads: 5, preloadindex: true, time: 12.21\n> maxthreads: 8, preloadindex: true, time: 12.14\n> maxthreads: 10, preloadindex: true, time: 12.08\n> maxthreads: 15, preloadindex: true, time: 12.16\n> maxthreads: 20, preloadindex: true, time: 11.96\n> \n> So it seams it gives me what preloadindex does, which is not much.\n> \n> Note: I'm not saying the patch is bad. I'm just wondering whether that\n> is expected.\n> \n> Michael\n> P.S.: It's actually sshfs with ssh to an NFSv3 client (server restricts\n> exports) :(\n"}]}