{"thread":{"id":"33147","subject":"[RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","startedAt":"2013-03-10T20:17:03Z","lastAt":"2013-03-25T11:13:14Z","messageCount":17,"participants":["Ramkumar Ramachandra","Heiko Voigt","Erik Faye-Lund","Jeff King","Karsten Blees","Duy Nguyen","Junio C Hamano","Pete Wyckoff","Thomas Rast"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"210986","messageId":"1362946623-23649-1-git-send-email-artagnon@gmail.com","threadId":"33147","inReplyTo":null,"subject":"[RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-10T20:17:03Z","receivedAt":"2013-03-10T20:17:03Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"git operations are slow on repositories with lots of files, and lots\nof tiny filesystem calls like lstat(), getdents(), open() are\nreposible for this. On the linux-2.6 repository, for instance, the\nnumbers for \"git status\" look like this:\n\n top syscalls sorted top syscalls sorted\n by acc. time by number\n ----------------------------------------------\n 0.401906 40950 lstat 0.401906 40950 lstat\n 0.190484 5343 getdents 0.150055 5374 open\n 0.150055 5374 open 0.190484 5343 getdents\n 0.074843 2806 close 0.074843 2806 close\n 0.003216 157 read 0.003216 157 read\n\nTo solve this problem, we propose to build a daemon which will watch\nthe filesystem using inotify and report batched up events over a UNIX\nsocket. Since inotify is Linux-only, we have to leave open the\npossibility of writing similar daemons for other platforms.\nEverything will continue to work as before if there is no helper\npresent.\n\nThe fswatch API introduces a generic way for git.git to request for\nfilesystem changes. Different helpers (like the inotify daemon on\nLinux) will be plugged into this API on different platforms. It falls\nback to using the filesystem calls.\n\nThe daemon will start up with the very first operation done on the git\nrepository, and will die after a specified period of repository\ninactivity. It is going to be a per-repo daemon and will write to a\nsocket in the repository: access control is managed by filesystem\npermissions.\n\nThis design is inspired by the credential helper design.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n Documentation/technical/api-fswatch.txt | 62 +++++++++++++++++++++++++++++++++\n 1 file changed, 62 insertions(+)\n create mode 100644 Documentation/technical/api-fswatch.txt\n\ndiff --git a/Documentation/technical/api-fswatch.txt b/Documentation/technical/api-fswatch.txt\nnew file mode 100644\nindex 0000000..9c6826a\n--- /dev/null\n+++ b/Documentation/technical/api-fswatch.txt\n@@ -0,0 +1,62 @@\n+fswatch API\n+===========\n+\n+The fswatch API provides an abstracted way of collecting information\n+about filesystem changes. A remote helper is typically a daemon which\n+uses inotify to watch the filesystem, and this information is used by\n+git instead of making expensive system calls like lstat(), open().\n+\n+Typical setup\n+-------------\n+\n+------------\n++-----------------------+\n+| Git code (C) |--- requires information about fs changes\n+|.......................|\n+| C fswatch API |--- system calls ---> filesystem\n++-----------------------+\n+ ^ |\n+ | UNIX socket |\n+ | v\n++-----------------------+\n+| Git fswatch helper |--- daemon inotify-watching ---> filesystem\n++-----------------------+\n+------------\n+\n+The Git code will call the C API to obtain changes in filesystem\n+information. The API will itself call a configured helper (e.g. \"git\n+fswatch-notify\") which may run filesystem changes, if the remote\n+helper daemon was started in a previous invocation. If the daemon is\n+not already running, it is started, and the C API will fall back to\n+making expensive system calls.\n+\n+C API\n+-----\n+\n+The credential C API is meant to be called by Git code which needs\n+information aboutx filesystem changes. It is centered around an\n+object representing the changes the filesystem since the last\n+invocation.\n+\n+Data Structures\n+~~~~~~~~~~~~~~~\n+\n+`struct fschanges`::\n+\n+\tTODO\n+\n+\n+Functions\n+~~~~~~~~~\n+\n+TODO\n+\n+Example\n+~~~~~~~\n+\n+TODO\n+\n+fswatch Helpers\n+---------------\n+\n+TODO\n-- \n1.8.1.5\n"},{"id":"211060","messageId":"20130311170542.GB4167@sandbox-ub.fritz.box","threadId":"33147","inReplyTo":"1362946623-23649-1-git-send-email-artagnon@gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2013-03-11T17:05:49Z","receivedAt":"2013-03-11T17:05:49Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Mon, Mar 11, 2013 at 01:47:03AM +0530, Ramkumar Ramachandra wrote:\n> git operations are slow on repositories with lots of files, and lots\n> of tiny filesystem calls like lstat(), getdents(), open() are\n> reposible for this. On the linux-2.6 repository, for instance, the\n> numbers for \"git status\" look like this:\n> \n> top syscalls sorted top syscalls sorted\n> by acc. time by number\n> ----------------------------------------------\n> 0.401906 40950 lstat 0.401906 40950 lstat\n> 0.190484 5343 getdents 0.150055 5374 open\n> 0.150055 5374 open 0.190484 5343 getdents\n> 0.074843 2806 close 0.074843 2806 close\n> 0.003216 157 read 0.003216 157 read\n> \n> To solve this problem, we propose to build a daemon which will watch\n> the filesystem using inotify and report batched up events over a UNIX\n> socket. Since inotify is Linux-only, we have to leave open the\n> possibility of writing similar daemons for other platforms.\n> Everything will continue to work as before if there is no helper\n> present.\n\nWhile talking about platform independence. How about Windows? AFAIK\nthere are no file based sockets. How about using shared memory, thats\navailable, instead? It would greatly reduce the needed porting effort.\n\nSince operations on a lot of files is especially expensive on Windows it\nis one of the platforms that would profit the most from such a daemon.\n\nCheers Heiko\n"},{"id":"211106","messageId":"CALkWK0m+CS25=Z91y90z49g4GK+b7oFugiXoH0h0J6P-yQXRCQ@mail.gmail.com","threadId":"33147","inReplyTo":"20130311170542.GB4167@sandbox-ub.fritz.box","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-12T09:43:39Z","receivedAt":"2013-03-12T09:43:39Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Heiko Voigt wrote:\n> While talking about platform independence. How about Windows? AFAIK\n> there are no file based sockets. How about using shared memory, thats\n> available, instead? It would greatly reduce the needed porting effort.\n\nWhat about the git credential helper: it uses UNIX sockets, no? How\ndoes git-credential-winstore [1] work?\n\n[1]: https://github.com/anurse/git-credential-winstore\n"},{"id":"211107","messageId":"CABPQNSYT2DiMeMe+6ropa8O7YUzAVR_b4CD+75qw4fw6XAv+8w@mail.gmail.com","threadId":"33147","inReplyTo":"CALkWK0m+CS25=Z91y90z49g4GK+b7oFugiXoH0h0J6P-yQXRCQ@mail.gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2013-03-12T09:50:15Z","receivedAt":"2013-03-12T09:50:15Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, Mar 12, 2013 at 10:43 AM, Ramkumar Ramachandra\n wrote:\n> Heiko Voigt wrote:\n>> While talking about platform independence. How about Windows? AFAIK\n>> there are no file based sockets. How about using shared memory, thats\n>> available, instead? It would greatly reduce the needed porting effort.\n>\n> What about the git credential helper: it uses UNIX sockets, no? How\n> does git-credential-winstore [1] work?\n>\n> [1]: https://github.com/anurse/git-credential-winstore\n\nFirst, we have a proper credential helper for Windows in\ncontrib/credential/wincred these days. As the one who wrote that, we\ncommunicate using stdin/stdout. The credential-helper doesn't maintain\nstate in itself, the Windows Credential Manager does. I suspect\ngit-credential-winstore works the same way.\n\nAs for Windows support, AFAIK there is no support for Unix domain\nsockets in Windows. But there is support for named pipes, which is\nalmost the same thing. What we have support for in compat/mingw.[ch]\nis a different matter, but we can extend that if needed.\n"},{"id":"211108","messageId":"20130312095519.GA12170@sigill.intra.peff.net","threadId":"33147","inReplyTo":"CALkWK0m+CS25=Z91y90z49g4GK+b7oFugiXoH0h0J6P-yQXRCQ@mail.gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-03-12T09:55:19Z","receivedAt":"2013-03-12T09:55:19Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 12, 2013 at 03:13:39PM +0530, Ramkumar Ramachandra wrote:\n\n> Heiko Voigt wrote:\n> > While talking about platform independence. How about Windows? AFAIK\n> > there are no file based sockets. How about using shared memory, thats\n> > available, instead? It would greatly reduce the needed porting effort.\n> \n> What about the git credential helper: it uses UNIX sockets, no? How\n> does git-credential-winstore [1] work?\n\nNo, the main credential protocol happens over pipes to a child process's\nstdin/stdout. The credential-cache helper does use unix sockets (since\nit needs to contact a long-running daemon that caches the credentials),\nand AFAIK is not available under Windows (but that's OK, because\nWindows-specific helpers that use secure storage are better anyway).\n\nWhen I introduced credential-cache, I recall somebody mentioned that\nthere is some Windows-equivalent IPC that can be used to emulate unix\ndomain sockets. The calls aren't the same, but as long as your\nrequirements are basically \"get messages to/from the daemon\", you can\nprobably abstract away the details on a per-platform basis.\n\nUnfortunately I can't seem to find the original message or any details\nin the archive (and I know next to nothing about Windows IPC).\n\n-Peff\n"},{"id":"211176","messageId":"513FB85C.5010106@gmail.com","threadId":"33147","inReplyTo":"1362946623-23649-1-git-send-email-artagnon@gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Karsten Blees","fromEmail":"karsten.blees@gmail.com","sentAt":"2013-03-12T23:21:00Z","receivedAt":"2013-03-12T23:21:00Z","isPatch":true,"sender":{"key":"karsten.blees@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1111200?v=4"},"body":"Am 10.03.2013 21:17, schrieb Ramkumar Ramachandra:\n> git operations are slow on repositories with lots of files, and lots\n> of tiny filesystem calls like lstat(), getdents(), open() are\n> reposible for this. On the linux-2.6 repository, for instance, the\n> numbers for \"git status\" look like this:\n> \n> top syscalls sorted top syscalls sorted\n> by acc. time by number\n> ----------------------------------------------\n> 0.401906 40950 lstat 0.401906 40950 lstat\n> 0.190484 5343 getdents 0.150055 5374 open\n> 0.150055 5374 open 0.190484 5343 getdents\n> 0.074843 2806 close 0.074843 2806 close\n> 0.003216 157 read 0.003216 157 read\n> \n> To solve this problem, we propose to build a daemon which will watch\n> the filesystem using inotify and report batched up events over a UNIX\n> socket.\n\n[...]\n\n> +\n> +The credential C API is meant to be called by Git code which needs\n> +information aboutx filesystem changes. It is centered around an\n> +object representing the changes the filesystem since the last\n> +invocation.\n> +\n\nHmmm...I don't see how filesystem changes since last invocation can solve the problem, or am I missing something? I think what you mean to say is that the daemon should keep track of the filesystem *state* of the working copy, or alternatively the deltas/changes to some known state (such as .git/index)?\n\nI'm also still skeptical whether a daemon will improve overall performance. In my understanding its essentially a filesystem cache in user-mode. The difference to using the OS filesystem cache directly (via lstat/readdir) is that we replace ~50k sys-calls with a single IPC call (i.e. the git <--> fswatch daemon communication is less 'chatty'). However, the 'chattyness' is still there between the fswatch daemon and the OS / inotify. Consider 'git status; make; make clean; git status'...that's a *lot* of changes to process for nothing (potentially slowing down make).\n\nThen there's the issue of stale data in the cache. Modifying porcelain commands that use 'git status --porcelain' to compile their changesets will want 100% exact data. I'm not saying its not doable, but adding another platform specific, caching daemon to the tool chain doesn't exactly simplify things...\n\nBut perhaps I'm too pessimistic (or just stigmatized by inherently slow and out-of-date TGitCache/TSvnCache on Windows :-)\n"},{"id":"211184","messageId":"CACsJy8CBru+Z0+oYVKGdwjiF4DDH3w4vCjunaoCnoDQ6AizwWg@mail.gmail.com","threadId":"33147","inReplyTo":"513FB85C.5010106@gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-03-13T01:03:32Z","receivedAt":"2013-03-13T01:03:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 13, 2013 at 6:21 AM, Karsten Blees wrote:\n> Hmmm...I don't see how filesystem changes since last invocation can solve the problem, or am I missing something? I think what you mean to say is that the daemon should keep track of the filesystem *state* of the working copy, or alternatively the deltas/changes to some known state (such as .git/index)?\n\nI think git process can keep track of filesystem state (and save it\ndown if necessary). But when git process is not running, system state\nchanges and it cannot know about. The daemon helps filling this gap\n(and basically keeps git \"running\" (in a light form) throughout a\ndevelopment session). For example if we know only 5 files have changed\nsince the last refresh, we only need to re-stat those 5. The same for\nuntracked/ignored file checking,\n\n> I'm also still skeptical whether a daemon will improve overall performance. In my understanding its essentially a filesystem cache in user-mode. The difference to using the OS filesystem cache directly (via lstat/readdir) is that we replace ~50k sys-calls with a single IPC call (i.e. the git <--> fswatch daemon communication is less 'chatty'). However, the 'chattyness' is still there between the fswatch daemon and the OS / inotify.\n\nI think it attempts to reduce unnecessary system calls, not eliminate\nthem all. In the \"5 changed files\" above, a few IPC calls are done to\nretrieve the file list, then 5 lstat will be issued (by git, not the\ndaemon) instead of thousands of them.\n\n>Consider 'git status; make; make clean; git status'...that's a *lot* of changes to process for nothing (potentially slowing down make).\n\nYeah. In my opinion, the daemon should realize that at some point\naccumulated changes are too much that it's not worth collecting\nanymore, and drop them all. Git will do it the normal/slow way. After\nthat the daemon picks up again. We only optimize for the case when\nlittle changes are made in filesystem.\n\n> Then there's the issue of stale data in the cache. Modifying porcelain commands that use 'git status --porcelain' to compile their changesets will want 100% exact data. I'm not saying its not doable, but adding another platform specific, caching daemon to the tool chain doesn't exactly simplify things...\n>\n> But perhaps I'm too pessimistic (or just stigmatized by inherently slow and out-of-date TGitCache/TSvnCache on Windows :-)\n\nThanks. I didn't know about TGitCache. Will dig it up. Maybe we can\nlearn something from it (or realize the daemon approach is futile\nafter all).\n-- \nDuy\n"},{"id":"211237","messageId":"5140BC80.4000201@gmail.com","threadId":"33147","inReplyTo":"CACsJy8CBru+Z0+oYVKGdwjiF4DDH3w4vCjunaoCnoDQ6AizwWg@mail.gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Karsten Blees","fromEmail":"karsten.blees@gmail.com","sentAt":"2013-03-13T17:50:56Z","receivedAt":"2013-03-13T17:50:56Z","isPatch":true,"sender":{"key":"karsten.blees@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1111200?v=4"},"body":"Am 13.03.2013 02:03, schrieb Duy Nguyen:\n> On Wed, Mar 13, 2013 at 6:21 AM, Karsten Blees wrote:\n>> Hmmm...I don't see how filesystem changes since last invocation can solve the problem, or am I missing something? I think what you mean to say is that the daemon should keep track of the filesystem *state* of the working copy, or alternatively the deltas/changes to some known state (such as .git/index)?\n> \n> I think git process can keep track of filesystem state (and save it\n> down if necessary).\n[...]\nAh, saving the state was the missing bits, thanks.\n\nHowever, AFAIK inotify doesn't work recursively, so the daemon would at least have to track the directory structure to be able to register / unregister inotify handlers as directories come and go.\n\n>> Consider 'git status; make; make clean; git status'...that's a *lot* of changes to process for nothing (potentially slowing down make).\n> \n> Yeah. In my opinion, the daemon should realize that at some point\n> accumulated changes are too much that it's not worth collecting\n> anymore, and drop them all. Git will do it the normal/slow way. After\n> that the daemon picks up again. We only optimize for the case when\n> little changes are made in filesystem.\n> \n\nThat sounds reasonable...\n\n>> Then there's the issue of stale data in the cache. Modifying porcelain commands that use 'git status --porcelain' to compile their changesets will want 100% exact data. I'm not saying its not doable, but adding another platform specific, caching daemon to the tool chain doesn't exactly simplify things...\n>>\n>> But perhaps I'm too pessimistic (or just stigmatized by inherently slow and out-of-date TGitCache/TSvnCache on Windows :-)\n> \n> Thanks. I didn't know about TGitCache. Will dig it up. Maybe we can\n> learn something from it (or realize the daemon approach is futile\n> after all).\n> \n\nTGitCache/TSvnCache are the background processes in TortoiseGit/TortoiseSvn that keep track of filesystem state to display icon overlays in Windows Explorer.\n"},{"id":"211244","messageId":"7vtxof146d.fsf@alter.siamese.dyndns.org","threadId":"33147","inReplyTo":"5140BC80.4000201@gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-13T19:38:02Z","receivedAt":"2013-03-13T19:38:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karsten Blees writes:\n\n> However, AFAIK inotify doesn't work recursively, so the daemon\n> would at least have to track the directory structure to be able to\n> register / unregister inotify handlers as directories come and go.\n\nYes, and you would need one inotify per directory but you do not\nhave an infinite supply of outstanding inotify watch (wasn't the\nlimit like 8k per a single uid or something?), so the daemon must be\nprepared to say \"I'll watch this, that and that directories, but the\nconsumers should check other directories themselves.\"\n\nFWIW, I share your suspicion that an effort in the direction this\nthread suggests may end up duplicating what the caching vfs layer\nalready does, and doing so poorly.\n"},{"id":"211285","messageId":"CACsJy8BOW-B0_gaNyL5dfdD-eKyknQEvEvf2nNy7xDWr=G02bQ@mail.gmail.com","threadId":"33147","inReplyTo":"7vtxof146d.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-03-14T10:58:18Z","receivedAt":"2013-03-14T10:58:18Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Mar 14, 2013 at 2:38 AM, Junio C Hamano wrote:\n> Karsten Blees writes:\n>\n>> However, AFAIK inotify doesn't work recursively, so the daemon\n>> would at least have to track the directory structure to be able to\n>> register / unregister inotify handlers as directories come and go.\n>\n> Yes, and you would need one inotify per directory but you do not\n> have an infinite supply of outstanding inotify watch (wasn't the\n> limit like 8k per a single uid or something?), so the daemon must be\n> prepared to say \"I'll watch this, that and that directories, but the\n> consumers should check other directories themselves.\"\n\nHey I did not know that. Webkit has about 6k leaf dirs and 182k files.\nWatching the top N biggest directories would cover M% of cached files:\n\n N M%\n 10 8.60\n 20 13.28\n 30 17.52\n 40 20.52\n 50 23.55\n 200 49.70\n 676 75.00\n 863 80.00\n1486 90.00\n\nSo it's trade-off. We can cut some syscall cost off but we probably\nneed to pay some for inotify. And we definitely can't watch full\nworktree. I don't know how costly it may be for watching many\ndirectories. If it's not so costly, watching 256 or 512 dirs might be\nenough.\n\nWhat about Windows? Does the equivalent mechanism have similar limits?\n\n> FWIW, I share your suspicion that an effort in the direction this\n> thread suggests may end up duplicating what the caching vfs layer\n> already does, and doing so poorly.\n\nI'm still curious how it works out. Maybe it's not up to the original\nexpectation, but hopefully it will speed things up a bit.\n-- \nDuy\n"},{"id":"211405","messageId":"20130315162717.GA12137@padd.com","threadId":"33147","inReplyTo":"7vtxof146d.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-03-15T16:27:17Z","receivedAt":"2013-03-15T16:27:17Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"gitster@pobox.com wrote on Wed, 13 Mar 2013 12:38 -0700:\n> Karsten Blees writes:\n> \n> > However, AFAIK inotify doesn't work recursively, so the daemon\n> > would at least have to track the directory structure to be able to\n> > register / unregister inotify handlers as directories come and go.\n> \n> Yes, and you would need one inotify per directory but you do not\n> have an infinite supply of outstanding inotify watch (wasn't the\n> limit like 8k per a single uid or something?), so the daemon must be\n> prepared to say \"I'll watch this, that and that directories, but the\n> consumers should check other directories themselves.\"\n\nfanotify is an option here too; it can watch an entire file\nsystem.\n\n\t\t-- Pete\n"},{"id":"211462","messageId":"87ehffv30f.fsf@pctrast.inf.ethz.ch","threadId":"33147","inReplyTo":"7vtxof146d.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2013-03-16T14:21:52Z","receivedAt":"2013-03-16T14:21:52Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Junio C Hamano writes:\n\n> Karsten Blees writes:\n>\n>> However, AFAIK inotify doesn't work recursively, so the daemon\n>> would at least have to track the directory structure to be able to\n>> register / unregister inotify handlers as directories come and go.\n>\n> Yes, and you would need one inotify per directory but you do not\n> have an infinite supply of outstanding inotify watch (wasn't the\n> limit like 8k per a single uid or something?), so the daemon must be\n> prepared to say \"I'll watch this, that and that directories, but the\n> consumers should check other directories themselves.\"\n\nThose are tunable limits though. For example I run this silly hack\n\n https://github.com/trast/watch\n\nwith the shell snippets to be able to quickly cd a shell to where\nsomething recently happened. I am able to watch most of my \"working\nset\" even under default limits, which here (opensuse tumbleweed, kernel\n3.8.x, x86_64) are\n\n $ cat /proc/sys/fs/inotify/max_user_watches \n 65536\n $ cat /proc/sys/fs/inotify/max_user_instances \n 128\n\nI'm not sure if other distros impose tighter limits by default, but as\nit stands you're not very likely to hit the 65k watches limit in any\ngiven repo. It seems more likely that you might hit the 128 instances\nlimit if we go with a design that uses one daemon per repo, if you run a\nscript that accesses many repos. For example, in an android tree I have\nlying around,\n\n $ repo list | wc -l\n 297\n\nThat alone might indicate it would be a good idea to have one \"global\"\ngit-agent that starts on demand, rather than a per-repo daemon.\nOtherwise we'd have to find a way to discover \"old\" daemons and tell\nthem to quit when we hit max_user_instances.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"211551","messageId":"CALkWK0=0+HYn=bSrGC5sMQOE-53As0h9dG3N9wvUB2=NW3=98A@mail.gmail.com","threadId":"33147","inReplyTo":"87ehffv30f.fsf@pctrast.inf.ethz.ch","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-18T08:24:03Z","receivedAt":"2013-03-18T08:24:03Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> Yes, and you would need one inotify per directory but you do not\n> have an infinite supply of outstanding inotify watch (wasn't the\n> limit like 8k per a single uid or something?), so the daemon must be\n> prepared to say \"I'll watch this, that and that directories, but the\n> consumers should check other directories themselves.\"\n>\n> FWIW, I share your suspicion that an effort in the direction this\n> thread suggests may end up duplicating what the caching vfs layer\n> already does, and doing so poorly.\n\nThomas Rast wrote:\n> $ cat /proc/sys/fs/inotify/max_user_watches\n> 65536\n> $ cat /proc/sys/fs/inotify/max_user_instancest\n> 128\n\n>From Junio's and Thomas' observations, I'm inclined to think that\ninotify is ill-suited for the problem we are trying to solve. It is\ndesigned as a per-directory watch, because VFS can quickly supply the\ninodes for a directory entry. As such, I think the ideal usecase for\ninotify is to execute something immediately when a change takes place\nin a directory: it's well-suited for solutions like Dropbox (which I\nthink is poorly designed to begin with, but that's offtopic). It\ndoesn't substitute of augment VFS caching. I suspect the VFS cache\nworks by caching the inodes in a frequently used directory entry, thus\noptimizing calls like lstat() on them.\n\nThe correct solution for our problem is to get VFS to recognize our\nrepository as a unit: the repository is not a bunch of frequently-used\ndirectory entries, but a frequently-used unit in itself. We need an\noptimization that will work on recursively on a directory entry.\nHowever, since the repository is a special usecase, I suspect adding\nan rwatch() system call (or similar) will be necessary to register the\nrepository with VFS. The design of this feature should be transparent\nto userland, and their filesystem calls will be optimized magically.\nWe certainly don't need something as fine-grained as inotify to\nperform these optimizations: if the tree hash of a registered\nrepository changes frequently enough, we have to optimize operations\non that directory tree (recursively).\n\nInputs from btrfs/ vfs hackers would be appreciated. I'll take out\nsome time to look at them myself this week.\n"},{"id":"211555","messageId":"87r4jdt404.fsf@pctrast.inf.ethz.ch","threadId":"33147","inReplyTo":"CALkWK0=0+HYn=bSrGC5sMQOE-53As0h9dG3N9wvUB2=NW3=98A@mail.gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-03-18T10:07:55Z","receivedAt":"2013-03-18T10:07:55Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Junio C Hamano wrote:\n>> Yes, and you would need one inotify per directory but you do not\n>> have an infinite supply of outstanding inotify watch (wasn't the\n>> limit like 8k per a single uid or something?), so the daemon must be\n>> prepared to say \"I'll watch this, that and that directories, but the\n>> consumers should check other directories themselves.\"\n>>\n>> FWIW, I share your suspicion that an effort in the direction this\n>> thread suggests may end up duplicating what the caching vfs layer\n>> already does, and doing so poorly.\n>\n> Thomas Rast wrote:\n>> $ cat /proc/sys/fs/inotify/max_user_watches\n>> 65536\n>> $ cat /proc/sys/fs/inotify/max_user_instancest\n>> 128\n>\n> From Junio's and Thomas' observations, I'm inclined to think that\n> inotify is ill-suited for the problem we are trying to solve. It is\n> designed as a per-directory watch, because VFS can quickly supply the\n> inodes for a directory entry. As such, I think the ideal usecase for\n> inotify is to execute something immediately when a change takes place\n> in a directory: it's well-suited for solutions like Dropbox (which I\n> think is poorly designed to begin with, but that's offtopic). It\n> doesn't substitute of augment VFS caching. I suspect the VFS cache\n> works by caching the inodes in a frequently used directory entry, thus\n> optimizing calls like lstat() on them.\n\nI have three objections to changing the kernel to fit us, as opposed to\njust using inotify:\n\n* inotify works. I can watch most of my $HOME with the hack I linked\n earlier[1]. Yes, it's a lot of coding around the problem that it is\n nonrecursive, but we already have a lot of code around the problem\n that we can't ask the VFS for diffs between points in time (namely,\n the whole business with an index and lstat() loops).\n\n* inotify is here today. Even if you got a hypothetical notifier into\n the kernel today, you'd have to wait months/years until it is\n available in distros, and years until everyone has it.\n\n* I'll bet you a beer that the kernel folks already had the same\n discussion when they made inotify. There has to be a reason why it's\n better than providing for recursive watches.\n\n\n[1] https://github.com/trast/watch\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"212159","messageId":"CALkWK0nt22NW=_iH2u8iiOMofoF6k+1sZLp-NsoYZtbADfCEuQ@mail.gmail.com","threadId":"33147","inReplyTo":"87r4jdt404.fsf@pctrast.inf.ethz.ch","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-25T10:44:24Z","receivedAt":"2013-03-25T10:44:24Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Just a small heads-up for people using Emacs. 24.4 has inotify\nsupport, and magit-inotify.el [1] has already started using it. From\ninitial impressions, I'm quite impressed with it.\n\n[1]: https://github.com/magit/magit/blob/master/contrib/magit-inotify.el\n"},{"id":"212162","messageId":"CACsJy8Bx83P=HK1pcy7ASnP5t+_snERxnVxji9_TGW-3fuUq-w@mail.gmail.com","threadId":"33147","inReplyTo":"CALkWK0nt22NW=_iH2u8iiOMofoF6k+1sZLp-NsoYZtbADfCEuQ@mail.gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-03-25T10:59:15Z","receivedAt":"2013-03-25T10:59:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 25, 2013 at 5:44 PM, Ramkumar Ramachandra\n wrote:\n> Just a small heads-up for people using Emacs. 24.4 has inotify\n> support, and magit-inotify.el [1] has already started using it. From\n> initial impressions, I'm quite impressed with it.\n\nHave you tried it? From a quick look, it seems to watch all\ndirectories. I wonder how it performs on webkit (at least 5k dirs)\n\n> [1]: https://github.com/magit/magit/blob/master/contrib/magit-inotify.el\n-- \nDuy\n"},{"id":"212164","messageId":"CALkWK0=f01C_5UP7WhgS0sXHdetbXe9YSroXx-PVvmp29FO9ew@mail.gmail.com","threadId":"33147","inReplyTo":"CACsJy8Bx83P=HK1pcy7ASnP5t+_snERxnVxji9_TGW-3fuUq-w@mail.gmail.com","subject":"Re: [RFC/PATCH] Documentation/technical/api-fswatch.txt: start with outline","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-25T11:13:14Z","receivedAt":"2013-03-25T11:13:14Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Duy Nguyen wrote:\n> On Mon, Mar 25, 2013 at 5:44 PM, Ramkumar Ramachandra\n> wrote:\n>> Just a small heads-up for people using Emacs. 24.4 has inotify\n>> support, and magit-inotify.el [1] has already started using it. From\n>> initial impressions, I'm quite impressed with it.\n>\n> Have you tried it? From a quick look, it seems to watch all\n> directories. I wonder how it performs on webkit (at least 5k dirs)\n\nYeah, but only on some small repositories. I expect it to be\nproblematic on big repositories: if I'm not mistaken, Emacs will\nblock.\n"}]}