threads / discuss / 35748

inotify support, nearly there

Subject: inotify support, nearly there

## tl;dr

6 messages between Jan 29, 2014 and Jul 22, 2014.

replies: 5people: 4as markdown or json

Duy Nguyen· Jan 29, 2014, 06:47 UTC · lore

Just a quick update for the enthusiasts. My branch file-watcher [1] has got working per-user inotify support. It's a 20 patch series so I'll refrain from spamming git@vger for a while, even though it hurts your eyes a lot less than what I have posted so far. The test suite ran fine with it so it's not that buggy. It has new tests too, even though real inotify is not tested in the new tests. Documentation is there, either in .txt or comments. Using it is simple:

$ mkdir ~/.watcher $ git file-watcher --detach ~/.watcher $ git config --global filewatcher.path $HOME/.watcher

There's still some polishing work to do. But I think the core logic is done. I have some ideas what to be polished, but I'd appreciate feedback if anyone uses it. We may need to make lookup code faster later.

MacOS, FreeBSD and Windows contributors. If you have time and are interested, have a look at the protocol, which is basically documented in file-watcher.c:handle_command(), and see if something is incompatible with your file notification mechanism. MacOS and FreeBSD may reuse code from file-watcher.c, at least the unix socket part. I'm not so sure about Windows. It probably needs a brand new daemon because little could be shared, I don't know. I deliberately design the daemon dumb so writing a completely new one won't be so hard. My plan is focus on inotify and get it merged first, then new OS support can come later (with refactoring if needed, but should not change the protocol drastically).

[1] git clone https://github.com/pclouds/git.git file-watcher -- Duy

Mike Hommey· Jan 29, 2014, 07:44 UTC · re: Duy Nguyen · lore

Re: inotify support, nearly there

On Wed, Jan 29, 2014 at 01:47:30PM +0700, Duy Nguyen wrote:

Show 30 quoted lines

> Just a quick update for the enthusiasts. My branch file-watcher [1]
> has got working per-user inotify support. It's a 20 patch series so
> I'll refrain from spamming git@vger for a while, even though it hurts
> your eyes a lot less than what I have posted so far. The test suite
> ran fine with it so it's not that buggy. It has new tests too, even
> though real inotify is not tested in the new tests. Documentation is
> there, either in .txt or comments. Using it is simple:
> 
> $ mkdir ~/.watcher
> $ git file-watcher --detach ~/.watcher
> $ git config --global filewatcher.path $HOME/.watcher
> 
> There's still some polishing work to do. But I think the core logic is
> done. I have some ideas what to be polished, but I'd appreciate
> feedback if anyone uses it. We may need to make lookup code faster
> later.
> 
> MacOS, FreeBSD and Windows contributors. If you have time and are
> interested, have a look at the protocol, which is basically documented
> in file-watcher.c:handle_command(), and see if something is
> incompatible with your file notification mechanism. MacOS and FreeBSD
> may reuse code from file-watcher.c, at least the unix socket part. I'm
> not so sure about Windows. It probably needs a brand new daemon
> because little could be shared, I don't know. I deliberately design
> the daemon dumb so writing a completely new one won't be so hard. My
> plan is focus on inotify and get it merged first, then new OS support
> can come later (with refactoring if needed, but should not change the
> protocol drastically).
> 
> [1] git clone https://github.com/pclouds/git.git file-watcher

Haven't looked at the code, so I don't know if you've done that, but in case you haven't, it would be nice to have an environment variable or a config option to make git use the file-watcher *and* normal lstat operations, to check consistency.

Mike

Duy Nguyen· Jan 29, 2014, 08:33 UTC · re: Mike Hommey · lore

Re: inotify support, nearly there

On Wed, Jan 29, 2014 at 2:44 PM, Mike Hommey <mh@glandium.org> wrote:

> Haven't looked at the code, so I don't know if you've done that, but in
> case you haven't, it would be nice to have an environment variable or a
> config option to make git use the file-watcher *and* normal lstat
> operations, to check consistency.

No I haven't. So far I depend on my manual tests and the test suite to check for consistency. Will do.

-- 
Duy

Juan P· Jul 21, 2014, 05:10 UTC · re: Duy Nguyen · lore

Re: inotify support, nearly there

Duy Nguyen <pclouds <at> gmail.com> writes:

Show 13 quoted lines

> 
> Just a quick update for the enthusiasts. My branch file-watcher [1]
> has got working per-user inotify support. It's a 20 patch series so
> I'll refrain from spamming git <at> vger for a while, even though it hurts
> your eyes a lot less than what I have posted so far. The test suite
> ran fine with it so it's not that buggy. It has new tests too, even
> though real inotify is not tested in the new tests. Documentation is
> there, either in .txt or comments. Using it is simple:
> 
> $ mkdir ~/.watcher
> $ git file-watcher --detach ~/.watcher
> $ git config --global filewatcher.path $HOME/.watcher
>

Will this mean that Git would work faster with repositories with large number of files or commits? I am new into this topic, but I am trying to understand, any pointers are appreciated.

Ronnie Sahlberg· Jul 22, 2014, 16:39 UTC · re: Juan P · lore

Re: inotify support, nearly there

On Sun, Jul 20, 2014 at 10:10 PM, Juan P <juandavid1707@gmail.com> wrote:

Show 20 quoted lines

> Duy Nguyen <pclouds <at> gmail.com> writes:
>
>>
>> Just a quick update for the enthusiasts. My branch file-watcher [1]
>> has got working per-user inotify support. It's a 20 patch series so
>> I'll refrain from spamming git <at> vger for a while, even though it hurts
>> your eyes a lot less than what I have posted so far. The test suite
>> ran fine with it so it's not that buggy. It has new tests too, even
>> though real inotify is not tested in the new tests. Documentation is
>> there, either in .txt or comments. Using it is simple:
>>
>> $ mkdir ~/.watcher
>> $ git file-watcher --detach ~/.watcher
>> $ git config --global filewatcher.path $HOME/.watcher
>>
>
> Will this mean that Git would work faster with repositories with large
> number of files or commits? I am new into this topic, but I am trying to
> understand, any pointers are appreciated.
>

That is the idea. Instead of having to recursively lstat() every file to find which ones have changed this will allow to query a database for "tell me all files that match this criteria xyz" which could speed up many things.

Duy, keep up the good work. Once you are ready I am more than happy to help out reviewing the patches.

regards ronnie sahlberg

Duy Nguyen· Jul 22, 2014, 23:27 UTC · re: Ronnie Sahlberg · lore

Re: inotify support, nearly there

On Tue, Jul 22, 2014 at 11:39 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:

> Duy, keep up the good work.
> Once you are ready I am more than happy to help out reviewing the patches.

The road I'm taking is:

1) split-index to speed up index writing time
2) add index-helper daemon to speed up index reading time
3) speed up .gitignore processing by caching the results and validate
using directory stat info
4) make sure 3) works (e.g. index extension data..) with another
source than directory stat info, like inotify
5) finally put inotify in place to kill lstat in index refresh and .gitignore

1) is entering next (or master) now. I need to submit 2) again and get
Windows guys' opinion (up to 3 it should work on Windows). As you can
see it'll like take a lot more months to get to 5). I feel a bit
guilty of "blocking" David Turner's inotify support series, which does
not require so many changes like this (but only benefits *nix),
because I _think_ this road is better. If you have time, maybe you can
have a look at his series and push it forward?

-- 
Duy