Re: [RFC] Speed up "git status" by caching untracked file info

From: Duy Nguyen <pclouds@gmail.com>
Date: Apr 22, 2014, 10:35 UTC
Message-ID: <CACsJy8AFDquHEajJQa_LWhtAMO-vHQbMCFuPV6UkGNZXTbANZw@mail.gmail.com>
In-Reply-To: <CACsJy8AYyhZKBmL=z2H11Oemjw_YoUES0vYtB8s0HnMB+4uyVw@mail.gmail.com>

On Tue, Apr 22, 2014 at 5:13 PM, Duy Nguyen <pclouds@gmail.com> wrote:

>> IIRC name_hash.c::lazy_init_name_hash took ~100ms on my system, so hopefully you did a dummy 'cache_name_exists("anything")' before starting the measurement of the first run?
>
> No I didn't. Thanks for pointing it out. I'll see if I can reduce its time.

Well name-hash is only used when core.ignorecase is set. So it's optional. Maybe we could save it in a separate index extension, but we need to verify that the reader uses the same hash function as the writer.

>> Similarly, the '--directory' option controls early returns from the directory scan (via read_directory_recursive's check_only argument), so you won't be able to get a full untracked files listing if the cache was recorded with '--directory'. Additionally, '--directory' aggregates the state at the topmost untracked directory, so that directory's cached state depends on all sub-directories as well...
>
> I missed this. We could ignore check_only if caching is enabled, but
> that does not sound really good. Let me think about it more..

We could save "check_only" to the cache as well. This way we don't have to disable the check_only trick completely.

So we process a directory with check_only set, find one untracked entry and stop short. We store check_only value and the status ("found something") in addition to dir mtime. Next time we check the dir's mtime. If it matches and is called with check_only set, we know there is at least one untracked entry, that's enough to stop r_d_r and return early. If dir mtime does not match, or r_d_r is called without check_only, we ignore the cached data and fall back to opendir.

Sounds good?

-- 
Duy