Re: Watchman/inotify support and other ways to speed up git status

From: Duy Nguyen <pclouds@gmail.com>
Date: Oct 22, 2015, 07:29 UTC
Message-ID: <CACsJy8BjTZkeq_AhQ+3F99NW0HTK--A0z5u42Z50vz08=0VapQ@mail.gmail.com>
In-Reply-To: <CAP8UFD3Cd9SOh6EYwcx9hTVv7P24M5bEJRCYCT5Qgj=qPRJ8hw@mail.gmail.com>

On Thu, Oct 22, 2015 at 7:59 AM, Christian Couder <christian.couder@gmail.com> wrote:

Show 12 quoted lines

> Hi everyone,
>
> I am starting to investigate ways to speed up git status and other git
> commands for Booking.com (thanks to AEvar) and I'd be happy to discuss
> the current status or be pointed to relevant documentation or mailing
> list threads.
>
> From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I
> understand that the status is roughly the following:
>
> - instead of working on inotify support it's better to work on using a
> cross platform tool like Watchman

Definitely. Especially because watchman has recently gained (experimental?) Windows support

Show 9 quoted lines

> - instead of working on Watchman support it is better to work first on
> caching information in the index
>
> - git update-index --untracked-cache has been developed by Duy and
> others and merged to master in May 2015 to cache untracked status in
> the index; it is still considered experimental
>
> - git index-helper has been worked on by Duy but its status is not
> clear (at least to me)

My roadmap is speeding up index write speed (split index), then read speed (index-helper) and watchman can be run on top of the index-helper. Untracked cache solves another performance problem with .gitignore. All these four pieces are big and getting slowly into git.git.

The last piece is using watchman with untracked cache to kill the (small) last stream of lstat() calls. But we'll see if we actually need it.

> What are the possible/planned next steps in this area? improving
> --untracked-cache? git index-helper? watchman support?

The index-helper needs some polishing and perhaps more eyeballing. I have the watchman patch on top of it, but I don't remember if I have ever sent it out. If anyone wants to help I can resend everything I have. I don't think I can resume the work soon.

-- 
Duy