Re: [PATCH 0/5] Use watchman to reduce index refresh time

From: Duy Nguyen <pclouds@gmail.com>
Date: Nov 3, 2015, 09:21 UTC
Message-ID: <CACsJy8Dz17gRSHch9e=iB2Kq2t4FbssatW84DF8pHTuFMgEgjQ@mail.gmail.com>
In-Reply-To: <CACsJy8DzSEVJYc85-3vSAZ8wB1pR9TLz0RrKyKFfHth9Tq+xyg@mail.gmail.com>

On Mon, Nov 2, 2015 at 8:23 PM, Duy Nguyen <pclouds@gmail.com> wrote:

Show 26 quoted lines

> On Mon, Nov 2, 2015 at 3:54 PM, Paolo Ciarrocchi
> <paolo.ciarrocchi@gmail.com> wrote:
>> On Sun, Nov 1, 2015 at 2:55 PM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
>>
>> Hi Duy,
>>
>>> This series builds on top of the index-helper series I just sent and
>>> uses watchman to keep track of file changes in order to avoid lstat()
>>> at refresh time. The series can also be found at [1]
>>>
>>> When I started this work, watchman did not support Windows yet. It
>>> does now, even if still experimental [2]. So Windows people, please
>>> try it out if you have time.
>>>
>>> To put all pieces so far together, we have split-index to reduce index
>>> write time, untracked cache to reduce I/O as well as computation for
>>> .gitignore, index-helper for index read time and this series for
>>> lstat() at refresh time. The remaining piece is killing lstat() from
>>> untracked cache, but right now it's just some idea and incomplete
>>> code.
>>
>> Did you manage to measure the speedup introduced by this series?
>
> It was from last year. I may have measured it but because I didn't
> save it in the commit message, it was lost anyway. Installing watchman
> and measuring with webkit.git soon..

Test repo: webkit.git with 104665 tracked files, 5615 untracked, 3517 dirs. Best numbers out of a few tries. This is best case scenario. Normal usage could have worse numbers.

There is something strange about the "-uno" measurements. I don't think watchman+untracked cache can beat -uno.. Maybe I did something wrong.

0m0.383s index v2 0m0.351s index v4 0m0.352s v2 split-index 0m0.309s v2 split index-helper 0m0.159s v2 split helper untracked-cache 0m0.123s v2 split helper "status -uno" 0m0.098s v2 split helper untracked watchman 0m0.071s v2 split helper watchman "status -uno"

Note, the watchman series needs s/free_watchman_shm/release_watchman_shm/ (I didn't do a good job of testing after rebase). And there's a small bug in index-helper --detach code writing incorrect PID..

-- 
Duy