Re: inotify to minimize stat() calls
- From
Torsten Bögershausen <tboegi@web.de>
- Date
- Mar 7, 2013, 22:16 UTC
- Message-ID
- <513911B3.7010903@web.de>
- In-Reply-To
- <CACsJy8DnvAjQPL4aP_LRC7aqx6OC4M5dMtj-OUot76qET2z08Q@mail.gmail.com>
On 11.02.13 03:56, Duy Nguyen wrote:
Show 41 quoted lines
> On Mon, Feb 11, 2013 at 3:16 AM, Junio C Hamano <gitster@pobox.com> wrote: >> The other "lstat()" experiment was a very interesting one, but this >> is not yet an interesting experiment to see where in the "ignore" >> codepath we are spending times. >> >> We know that we can tell wt_status_collect_untracked() not to bother >> with the untracked or ignored files with !s->show_untracked_files >> already, but I think the more interesting question is if we can show >> the untracked files with less overhead. >> >> If we want to show untrackedd files, it is a given that we need to >> read directories to see what paths there are on the filesystem. Is >> the opendir/readdir cost dominating in the process? Are we spending >> a lot of time sifting the result of opendir/readdir via the ignore >> mechanism? Is reading the "ignore" files costing us much to prime >> the ignore mechanism? >> >> If readdir cost is dominant, then that makes "cache gitignore" a >> nonsense proposition, I think. If you really want to "cache" >> something, you need to have somebody (i.e. a daemon) who constantly >> keeps an eye on the filesystem changes and can respond with the up >> to date result directly to fill_directory(). I somehow doubt that >> it is a direction we would want to go in, though. > > Yeah, it did not cut out syscall cost, I also cut a lot of user-space > processing (plus .gitignore content access). From the timings I posted > earlier, > >> unmodified dir.c >> real 0m0.550s 0m0.287s >> user 0m0.305s 0m0.201s >> sys 0m0.240s 0m0.084s > > sys time is reduced from 0.24s to 0.08s, so readdir+opendir definitely > has something to do with it (and perhaps reading .gitignore). But it > also reduces user time from 0.305 to 0.201s. I don't think avoiding > readdir+openddir will bring us this gain. It's probably the cost of > matching .gitignore. I'll try to replace opendir+readdir with a > no-syscall version. At this point "untracked caching" sounds more > feasible (and less complex) than ".gitignore cachine". >
Thanks for Duy for the measurements, and patches. I took the freedom to convert the patched dir.c into a "runtime configurable" git status option. I'm not sure if the following copy-and-paste work applies, (it is based on Git 1.8.1.3), but the time spend for "git status --changed-only" is basically half the time of "git status", similar to what Duy has measured. I did a test both on a Linux box and Mac OS.
And the speedup is so impressive, that I am tempted to submit a patch simlar to the following, what do we think about it? /Torsten
-- >8 --
[PATCH] git status: add option changed-only git status may be run faster if - we only check if files are changed which are already known to git. - we don't check if there are untracked files.
"git status --changed-only" (or the short form "git status -c")
will only check for changed files which are already known to git, and which are in the index.
The call to read_directory_recursive() is skipped and untracked files in the working tree are not reported.
Inspired-by: Duy Nguyen <pclouds@gmail.com> Signed-off-by: Torsten Bögershausen <tboegi@web.de> --- builtin/commit.c | 2 ++ dir.c | 5 +++-- dir.h | 3 ++- wt-status.c | 3 +++ wt-status.h | 1 + 5 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/builtin/commit.c b/builtin/commit.c index d6dd3df..6a5ba11 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1158,6 +1158,8 @@ int cmd_status(int argc, const char **argv, const char *prefix) unsigned char sha1[20]; static struct option builtin_status_options[] = { OPT__VERBOSE(&verbose, N_("be verbose")), + OPT_BOOLEAN('c', "changed-only", &s.check_changed_only, + N_("Ignore untracked files. Check if files known to git are modified")), OPT_SET_INT('s', "short", &status_format, N_("show status concisely"), STATUS_FORMAT_SHORT), OPT_BOOLEAN('b', "branch", &s.show_branch, diff --git a/dir.c b/dir.c index a473ca2..555b652 100644 --- a/dir.c +++ b/dir.c @@ -1274,8 +1274,9 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const char return dir->nr; simplify = create_simplify(pathspec); - if (!len || treat_leading_path(dir, path, len, simplify)) - read_directory_recursive(dir, path, len, 0, simplify); + if ((!(dir->flags & DIR_CHECK_CHANGED_ONLY)) && + (!len || treat_leading_path(dir, path, len, simplify))) o + read_directory_recursive(dir, path, len, 0, simplify); free_simplify(simplify); qsort(dir->entries, dir->nr, sizeof(struct dir_entry *), cmp_name); qsort(dir->ignored, dir->ignored_nr, sizeof(struct dir_entry *), cmp_name); diff --git a/dir.h b/dir.h index f5c89e3..1a915a7 100644 --- a/dir.h +++ b/dir.h @@ -41,7 +41,8 @@ struct dir_struct { DIR_SHOW_OTHER_DIRECTORIES = 1<<1, DIR_HIDE_EMPTY_DIRECTORIES = 1<<2, DIR_NO_GITLINKS = 1<<3, - DIR_COLLECT_IGNORED = 1<<4 + DIR_COLLECT_IGNORED = 1<<4, + DIR_CHECK_CHANGED_ONLY = 1<<5 } flags; struct dir_entry **entries; struct dir_entry **ignored; diff --git a/wt-status.c b/wt-status.c index d7cfe8f..b315785 100644 --- a/wt-status.c +++ b/wt-status.c @@ -503,6 +503,9 @@ static void wt_status_collect_untracked(struct wt_status *s) if (s->show_untracked_files != SHOW_ALL_UNTRACKED_FILES) dir.flags |= DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES; + if (s->check_changed_only) + dir.flags |= DIR_CHECK_CHANGED_ONLY; + setup_standard_excludes(&dir); fill_directory(&dir, s->pathspec); diff --git a/wt-status.h b/wt-status.h index 236b41f..7eb0115 100644 --- a/wt-status.h +++ b/wt-status.h @@ -47,6 +47,7 @@ struct wt_status { const char **pathspec; int verbose; int amend; + int check_changed_only; enum commit_whence whence; int nowarn; int use_color;
-- 1.8.2.rc2