Re: [RFC] On watchman support

From: Duy Nguyen <pclouds@gmail.com>
Date: Nov 18, 2014, 10:48 UTC
Message-ID: <CACsJy8BfxP7KF1XF29BOgC6XhO8iAy-ycEoLkDG5rn6TYH_DrA@mail.gmail.com>
In-Reply-To: <1416270336.13653.23.camel@leckie>

On Tue, Nov 18, 2014 at 7:25 AM, David Turner <dturner@twopensource.com> wrote:

Show 6 quoted lines

>> So we got a few options:
>>
>> 1) Convince watchman devs to add something to make it work
>
> Based on the thread on the watchman github it looks like this won't
> happen.

Yeah. I came to the conclusion that I needed an extra daemon. And because I would need an extra daemon anyway to speed up index read time, that one could be used for caching something else like watchman. It works out quite nice: because watchman is not tied to the main 'git' binary, people don't need libwatchman and libjansson by default. When people want watchman, they can install an extra package that includes the index-helper and its dependencies. This only matters for binary-based distros of course.

Show 7 quoted lines

>> Comments?
>
> I don't think it would be impossible to add Windows support to watchman;
> the necessary functions exist, although I don't know how well they work.
> My experience with watchman is that it is something of a stress test of
> a filesystem's notification layer.  It has exposed bugs in inotify, and
> caused system instability on OS X.

The way i'm adding watchman to index-helper should work on Windows as well, IPC is really simple. But let's wait and see.

Show 6 quoted lines

> My patches are not the world's most beautiful, but they do work.  I
> think some improvement might be possible by keeping info about tracked
> files in the index, and only storing the tree of ignored and untracked
> files separately.  But I have not thought this through fully.  In any
> case, making use of shared memory for the fs_cache (as some of your
> other patches do for the index) would definitely save time.

By the way, what happened to your sse optimization in refs.c? I see it's reverted but I didn't follow closely to know why. Or will you go with cityhash now.. I ask because you have another sse optimization for hashmap on your watchman branch and that could reduce init time for name-hash. Name-hash is used often on case-insensitive fs (less often on case-sensitive fs).

I did a simple test and your optimization could init name-hash (on webkit) in 35ms, while unmodified hashmap took 88ms. Loading index on this machine took 360ms for reference (probably down too 100ms with index-helper running, when that 88ms starts to become significant).

-- 
Duy