Re: inotify to minimize stat() calls

From: Karsten Blees <karsten.blees@gmail.com>
Date: Feb 19, 2013, 14:25 UTC
Message-ID: <51238B59.1050106@dcon.de>
In-Reply-To: <CALkWK0=XFBfZjO3oCJ8Jxya1ud79MQcQFm6pmZpOU8c3MxVtqQ@mail.gmail.com>

Am 19.02.2013 10:49, schrieb Ramkumar Ramachandra:

Show 16 quoted lines

> Karsten Blees wrote:
>> Am 11.02.2013 04:53, schrieb Duy Nguyen:
>>> On Sun, Feb 10, 2013 at 11:58 PM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
>>>> Karsten Blees has done something similar-ish on Windows, and he posted
>>>> the results here:
>>>>
>>>> https://groups.google.com/forum/#!topic/msysgit/fL_jykUmUNE/discussion
>>>>
>>
>> The new hashtable implementation in fscache [1] supports O(1) removal and has no mingw dependencies - might come in handy for anyone trying to implement an inotify daemon.
>>
>> [1] https://github.com/kblees/git/commit/f7eb85c2
> 
> Thanks!  I'm cherry-picking this.  Why didn't you propose replacing
> hash.{c,h} with this in git.git though?
>

I was planning to, but didn't find the time yet to adapt existing hash.[ch] uses to the new version, and there's not much use adding four more files of dead code. If someone else could jump in here that would be great.

Note that there's another t0007 now, so t/t0007-hashmap.sh needs to be renamed.

Show 15 quoted lines

>>>> I also seem to remember he doing a ReadDirectoryChangesW version, but
>>>> I don't remember what happened with that.
>>>
>>> Thanks. I came across that but did not remember. For one thing, we
>>> know the inotify alternative for Windows: ReadDirectoryChangesW.
>>>
>>
>> I dropped ReadDirectoryChangesW because maintaining a 'live' file system cache became more and more complicated. For example, according to MSDN docs, ReadDirectoryChangesW *may* report short DOS 8.3 names (i.e. "PROGRA~1" instead of "Program Files"), so a correct and fast cache implementation would have to be indexed by long *and* short names...
>>
>> Another problem was that the 'live' cache had quite negative performance impact on mutating git commands (checkout, reset...). An inotify daemon running as a background process (not in-process as fscache) will probably affect everyone that modifies the working copy, e.g. running 'make' or the test-suite. This should be considered in the design.
> 
> Yes, an external daemon would report creation of *.o files, from the
> compile, for instance.  We need a way for it to be filtered at the
> daemon itself, so git isn't burdened with the filtering.
>

...and this filtering should affect foreground processes as little as possible. For example, gaining 1 s per git-status is counter-productive if compile time increases by 10 s because the daemon re-reads .gitignore files for every new *.o.