# inotify support, nearly there 6 messages from 2014-01-29 to 2014-07-22. Participants: Duy Nguyen, Mike Hommey, Juan P, Ronnie Sahlberg. Thread: https://gitlist.dev/t/35748 ## Duy Nguyen, 2014-01-29 06:47 Subject: inotify support, nearly there Message-ID: URL: https://gitlist.dev/e/CACsJy8CG5QUqYOM46mOHOWDKB-A45B2-fj1uS9OtgniV%2BP8Ktg%40mail.gmail.com ``` Just a quick update for the enthusiasts. My branch file-watcher [1] has got working per-user inotify support. It's a 20 patch series so I'll refrain from spamming git@vger for a while, even though it hurts your eyes a lot less than what I have posted so far. The test suite ran fine with it so it's not that buggy. It has new tests too, even though real inotify is not tested in the new tests. Documentation is there, either in .txt or comments. Using it is simple: $ mkdir ~/.watcher $ git file-watcher --detach ~/.watcher $ git config --global filewatcher.path $HOME/.watcher There's still some polishing work to do. But I think the core logic is done. I have some ideas what to be polished, but I'd appreciate feedback if anyone uses it. We may need to make lookup code faster later. MacOS, FreeBSD and Windows contributors. If you have time and are interested, have a look at the protocol, which is basically documented in file-watcher.c:handle_command(), and see if something is incompatible with your file notification mechanism. MacOS and FreeBSD may reuse code from file-watcher.c, at least the unix socket part. I'm not so sure about Windows. It probably needs a brand new daemon because little could be shared, I don't know. I deliberately design the daemon dumb so writing a completely new one won't be so hard. My plan is focus on inotify and get it merged first, then new OS support can come later (with refactoring if needed, but should not change the protocol drastically). [1] git clone https://github.com/pclouds/git.git file-watcher -- Duy ``` ## Mike Hommey, 2014-01-29 07:44 Subject: Re: inotify support, nearly there Message-ID: <20140129074435.GA21044@glandium.org> URL: https://gitlist.dev/e/20140129074435.GA21044%40glandium.org In-Reply-To: ``` On Wed, Jan 29, 2014 at 01:47:30PM +0700, Duy Nguyen wrote: > Just a quick update for the enthusiasts. My branch file-watcher [1] > has got working per-user inotify support. It's a 20 patch series so > I'll refrain from spamming git@vger for a while, even though it hurts > your eyes a lot less than what I have posted so far. The test suite > ran fine with it so it's not that buggy. It has new tests too, even > though real inotify is not tested in the new tests. Documentation is > there, either in .txt or comments. Using it is simple: > > $ mkdir ~/.watcher > $ git file-watcher --detach ~/.watcher > $ git config --global filewatcher.path $HOME/.watcher > > There's still some polishing work to do. But I think the core logic is > done. I have some ideas what to be polished, but I'd appreciate > feedback if anyone uses it. We may need to make lookup code faster > later. > > MacOS, FreeBSD and Windows contributors. If you have time and are > interested, have a look at the protocol, which is basically documented > in file-watcher.c:handle_command(), and see if something is > incompatible with your file notification mechanism. MacOS and FreeBSD > may reuse code from file-watcher.c, at least the unix socket part. I'm > not so sure about Windows. It probably needs a brand new daemon > because little could be shared, I don't know. I deliberately design > the daemon dumb so writing a completely new one won't be so hard. My > plan is focus on inotify and get it merged first, then new OS support > can come later (with refactoring if needed, but should not change the > protocol drastically). > > [1] git clone https://github.com/pclouds/git.git file-watcher Haven't looked at the code, so I don't know if you've done that, but in case you haven't, it would be nice to have an environment variable or a config option to make git use the file-watcher *and* normal lstat operations, to check consistency. Mike ``` ## Duy Nguyen, 2014-01-29 08:33 Subject: Re: inotify support, nearly there Message-ID: URL: https://gitlist.dev/e/CACsJy8Bi-dFbWKOGw4C3fUXZ2p_0mc5CjhyMkL9p_F0H4V1rOg%40mail.gmail.com In-Reply-To: <20140129074435.GA21044@glandium.org> ``` On Wed, Jan 29, 2014 at 2:44 PM, Mike Hommey wrote: > Haven't looked at the code, so I don't know if you've done that, but in > case you haven't, it would be nice to have an environment variable or a > config option to make git use the file-watcher *and* normal lstat > operations, to check consistency. No I haven't. So far I depend on my manual tests and the test suite to check for consistency. Will do. -- Duy ``` ## Juan P, 2014-07-21 05:10 Subject: Re: inotify support, nearly there Message-ID: URL: https://gitlist.dev/e/loom.20140721T070730-175%40post.gmane.org In-Reply-To: ``` Duy Nguyen gmail.com> writes: > > Just a quick update for the enthusiasts. My branch file-watcher [1] > has got working per-user inotify support. It's a 20 patch series so > I'll refrain from spamming git vger for a while, even though it hurts > your eyes a lot less than what I have posted so far. The test suite > ran fine with it so it's not that buggy. It has new tests too, even > though real inotify is not tested in the new tests. Documentation is > there, either in .txt or comments. Using it is simple: > > $ mkdir ~/.watcher > $ git file-watcher --detach ~/.watcher > $ git config --global filewatcher.path $HOME/.watcher > Will this mean that Git would work faster with repositories with large number of files or commits? I am new into this topic, but I am trying to understand, any pointers are appreciated. ``` ## Ronnie Sahlberg, 2014-07-22 16:39 Subject: Re: inotify support, nearly there Message-ID: URL: https://gitlist.dev/e/CAL%3DYDWk90x0XBjj7rkSaeTZ-pwSj%3DvktvOYbyivpqqwJVAGMRw%40mail.gmail.com In-Reply-To: ``` On Sun, Jul 20, 2014 at 10:10 PM, Juan P wrote: > Duy Nguyen gmail.com> writes: > >> >> Just a quick update for the enthusiasts. My branch file-watcher [1] >> has got working per-user inotify support. It's a 20 patch series so >> I'll refrain from spamming git vger for a while, even though it hurts >> your eyes a lot less than what I have posted so far. The test suite >> ran fine with it so it's not that buggy. It has new tests too, even >> though real inotify is not tested in the new tests. Documentation is >> there, either in .txt or comments. Using it is simple: >> >> $ mkdir ~/.watcher >> $ git file-watcher --detach ~/.watcher >> $ git config --global filewatcher.path $HOME/.watcher >> > > Will this mean that Git would work faster with repositories with large > number of files or commits? I am new into this topic, but I am trying to > understand, any pointers are appreciated. > That is the idea. Instead of having to recursively lstat() every file to find which ones have changed this will allow to query a database for "tell me all files that match this criteria xyz" which could speed up many things. Duy, keep up the good work. Once you are ready I am more than happy to help out reviewing the patches. regards ronnie sahlberg ``` ## Duy Nguyen, 2014-07-22 23:27 Subject: Re: inotify support, nearly there Message-ID: URL: https://gitlist.dev/e/CACsJy8B%3DXLPOj0YNwdmmFtOaVBXDR1UN%2Bk-vz9M%2BnRsTGUHHxA%40mail.gmail.com In-Reply-To: ``` On Tue, Jul 22, 2014 at 11:39 PM, Ronnie Sahlberg wrote: > Duy, keep up the good work. > Once you are ready I am more than happy to help out reviewing the patches. The road I'm taking is: 1) split-index to speed up index writing time 2) add index-helper daemon to speed up index reading time 3) speed up .gitignore processing by caching the results and validate using directory stat info 4) make sure 3) works (e.g. index extension data..) with another source than directory stat info, like inotify 5) finally put inotify in place to kill lstat in index refresh and .gitignore 1) is entering next (or master) now. I need to submit 2) again and get Windows guys' opinion (up to 3 it should work on Windows). As you can see it'll like take a lot more months to get to 5). I feel a bit guilty of "blocking" David Turner's inotify support series, which does not require so many changes like this (but only benefits *nix), because I _think_ this road is better. If you have time, maybe you can have a look at his series and push it forward? -- Duy ```