{"thread":{"id":"40615","subject":"Watchman/inotify support and other ways to speed up git status","startedAt":"2015-10-22T05:59:29Z","lastAt":"2015-11-03T20:32:21Z","messageCount":8,"participants":["Christian Couder","Duy Nguyen","David Turner"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"272087","messageId":"CAP8UFD3Cd9SOh6EYwcx9hTVv7P24M5bEJRCYCT5Qgj=qPRJ8hw@mail.gmail.com","threadId":"40615","inReplyTo":null,"subject":"Watchman/inotify support and other ways to speed up git status","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2015-10-22T05:59:29Z","receivedAt":"2015-10-22T05:59:29Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi everyone,\n\nI am starting to investigate ways to speed up git status and other git\ncommands for Booking.com (thanks to AEvar) and I'd be happy to discuss\nthe current status or be pointed to relevant documentation or mailing\nlist threads.\n\n>From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I\nunderstand that the status is roughly the following:\n\n- instead of working on inotify support it's better to work on using a\ncross platform tool like Watchman\n\n- instead of working on Watchman support it is better to work first on\ncaching information in the index\n\n- git update-index --untracked-cache has been developed by Duy and\nothers and merged to master in May 2015 to cache untracked status in\nthe index; it is still considered experimental\n\n- git index-helper has been worked on by Duy but its status is not\nclear (at least to me)\n\nIs that correct?\nWhat are the possible/planned next steps in this area? improving\n--untracked-cache? git index-helper? watchman support?\n\nThanks,\nChristian.\n\n[0] March 8 2015: [PATCH 00/24] nd/untracked-cache updates\nhttp://thread.gmane.org/gmane.comp.version-control.git/265053/\n\n[1] November 11 2014: [RFC] On watchman support\nhttp://thread.gmane.org/gmane.comp.version-control.git/259399/\n\n[2] October 27 2014:[PATCH 00/19] Untracked cache to speed up \"git status\"\nhttp://thread.gmane.org/gmane.comp.version-control.git/258766\n\n[3] July 28 2014: [PATCH v3 0/9] Speed up cache loading time\nhttp://thread.gmane.org/gmane.comp.version-control.git/254314/\n\n[4] May 7 2014: [PATCH 00/20] Untracked cache to speed up \"git status\"\nhttp://thread.gmane.org/gmane.comp.version-control.git/248306\n\n[5] May 2 2014: Watchman support for git\nhttp://thread.gmane.org/gmane.comp.version-control.git/248004/\n\n[6] March 10 2014:\nhttp://git.661346.n2.nabble.com/question-about-Facebook-makes-Mercurial-faster-than-Git-tt7605273.html#a7605280\n\n[7] January 29 2014:\nhttp://git.661346.n2.nabble.com/inotify-support-nearly-there-tt7602739.html\n\n[8] January 12 2014:\nhttp://git.661346.n2.nabble.com/PATCH-0-6-inotify-support-tt7601877.html#a7603955\n"},{"id":"272088","messageId":"CACsJy8BjTZkeq_AhQ+3F99NW0HTK--A0z5u42Z50vz08=0VapQ@mail.gmail.com","threadId":"40615","inReplyTo":"CAP8UFD3Cd9SOh6EYwcx9hTVv7P24M5bEJRCYCT5Qgj=qPRJ8hw@mail.gmail.com","subject":"Re: Watchman/inotify support and other ways to speed up git status","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-10-22T07:29:57Z","receivedAt":"2015-10-22T07:29:57Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Oct 22, 2015 at 7:59 AM, Christian Couder\n<christian.couder@gmail.com> wrote:\n> Hi everyone,\n>\n> I am starting to investigate ways to speed up git status and other git\n> commands for Booking.com (thanks to AEvar) and I'd be happy to discuss\n> the current status or be pointed to relevant documentation or mailing\n> list threads.\n>\n> From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I\n> understand that the status is roughly the following:\n>\n> - instead of working on inotify support it's better to work on using a\n> cross platform tool like Watchman\n\nDefinitely. Especially because watchman has recently gained\n(experimental?) Windows support\n\n> - instead of working on Watchman support it is better to work first on\n> caching information in the index\n>\n> - git update-index --untracked-cache has been developed by Duy and\n> others and merged to master in May 2015 to cache untracked status in\n> the index; it is still considered experimental\n>\n> - git index-helper has been worked on by Duy but its status is not\n> clear (at least to me)\n\nMy roadmap is speeding up index write speed (split index), then read\nspeed (index-helper) and watchman can be run on top of the\nindex-helper. Untracked cache solves another performance problem with\n.gitignore. All these four pieces are big and getting slowly into\ngit.git.\n\nThe last piece is using watchman with untracked cache to kill the\n(small) last stream of lstat() calls. But we'll see if we actually\nneed it.\n\n> What are the possible/planned next steps in this area? improving\n> --untracked-cache? git index-helper? watchman support?\n\nThe index-helper needs some polishing and perhaps more eyeballing. I\nhave the watchman patch on top of it, but I don't remember if I have\never sent it out. If anyone wants to help I can resend everything I\nhave. I don't think I can resume the work soon.\n-- \nDuy\n"},{"id":"272364","messageId":"1445990089.8302.27.camel@twopensource.com","threadId":"40615","inReplyTo":"CAP8UFD3Cd9SOh6EYwcx9hTVv7P24M5bEJRCYCT5Qgj=qPRJ8hw@mail.gmail.com","subject":"Re: Watchman/inotify support and other ways to speed up git status","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-10-27T23:54:49Z","receivedAt":"2015-10-27T23:54:49Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"\nOn Thu, 2015-10-22 at 07:59 +0200, Christian Couder wrote:\n> Hi everyone,\n> \n> I am starting to investigate ways to speed up git status and other git\n> commands for Booking.com (thanks to AEvar) and I'd be happy to discuss\n> the current status or be pointed to relevant documentation or mailing\n> list threads.\n> \n> From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I\n> understand that the status is roughly the following:\n> \n> - instead of working on inotify support it's better to work on using a\n> cross platform tool like Watchman\n> \n> - instead of working on Watchman support it is better to work first on\n> caching information in the index\n> \n> - git update-index --untracked-cache has been developed by Duy and\n> others and merged to master in May 2015 to cache untracked status in\n> the index; it is still considered experimental\n> \n> - git index-helper has been worked on by Duy but its status is not\n> clear (at least to me)\n> \n> Is that correct?\n> What are the possible/planned next steps in this area? improving\n\nWe're using Watchman at Twitter.  A week or two ago posted a dump of our\ncode to github, but I would advise waiting a day or two to use it, as\nI'm about to pull a large number of bugfixes into it (I'll update this\nthread and provide a link once I do so).  \n\nIt's good, but it's not great.  One major problem is a bug on OS X[1]\nthat causes missed updates.  Another is that wide changes end up being\nquite inefficient when querying watchman.  This means that we do some\nhackery to manually update the fs_cache during various large git\noperations.\n\nI agree that in general it would be better to store or all some of this\ninformation in the index, and the untracked-cache is a good step on\nthat. But with it enabled and watchman disabled, there still appears to\nbe 1 lstat per file (plus one stat per dir).  The stats per-directory\nalone are a large issue for Twitter because we have a relatively deep\nand bushy directory structure (an average dir has about 3 or 4 entries\nin it).  As a result, git status with watchman is almost twice as fast\nas with the untracked cache (on my particular machine).\n\n\n[1] https://github.com/facebook/watchman/issues/172\n"},{"id":"272476","messageId":"CAP8UFD3rkacENsnthdhqTPczbZP+J_iV6xr8sTXj2MFgZRx8DQ@mail.gmail.com","threadId":"40615","inReplyTo":"1445990089.8302.27.camel@twopensource.com","subject":"Re: Watchman/inotify support and other ways to speed up git status","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2015-10-29T08:10:05Z","receivedAt":"2015-10-29T08:10:05Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Oct 28, 2015 at 12:54 AM, David Turner <dturner@twopensource.com> wrote:\n>\n> On Thu, 2015-10-22 at 07:59 +0200, Christian Couder wrote:\n>> Hi everyone,\n>>\n>> I am starting to investigate ways to speed up git status and other git\n>> commands for Booking.com (thanks to AEvar) and I'd be happy to discuss\n>> the current status or be pointed to relevant documentation or mailing\n>> list threads.\n>>\n>> From the threads below ([0], [1], [2], [3], [4], [5], [6], [7], [8]) I\n>> understand that the status is roughly the following:\n>>\n>> - instead of working on inotify support it's better to work on using a\n>> cross platform tool like Watchman\n>>\n>> - instead of working on Watchman support it is better to work first on\n>> caching information in the index\n>>\n>> - git update-index --untracked-cache has been developed by Duy and\n>> others and merged to master in May 2015 to cache untracked status in\n>> the index; it is still considered experimental\n>>\n>> - git index-helper has been worked on by Duy but its status is not\n>> clear (at least to me)\n>>\n>> Is that correct?\n>> What are the possible/planned next steps in this area? improving\n>\n> We're using Watchman at Twitter.  A week or two ago posted a dump of our\n> code to github, but I would advise waiting a day or two to use it, as\n> I'm about to pull a large number of bugfixes into it (I'll update this\n> thread and provide a link once I do so).\n\nGreat, I will have a look at it then!\n\n> It's good, but it's not great.  One major problem is a bug on OS X[1]\n> that causes missed updates.  Another is that wide changes end up being\n> quite inefficient when querying watchman.  This means that we do some\n> hackery to manually update the fs_cache during various large git\n> operations.\n>\n> I agree that in general it would be better to store or all some of this\n> information in the index, and the untracked-cache is a good step on\n> that. But with it enabled and watchman disabled, there still appears to\n> be 1 lstat per file (plus one stat per dir).  The stats per-directory\n> alone are a large issue for Twitter because we have a relatively deep\n> and bushy directory structure (an average dir has about 3 or 4 entries\n> in it).  As a result, git status with watchman is almost twice as fast\n> as with the untracked cache (on my particular machine).\n\nThanks for this detailled description.\n\n> [1] https://github.com/facebook/watchman/issues/172\n"},{"id":"272756","messageId":"1446497776.4131.6.camel@twopensource.com","threadId":"40615","inReplyTo":"CAP8UFD3rkacENsnthdhqTPczbZP+J_iV6xr8sTXj2MFgZRx8DQ@mail.gmail.com","subject":"Re: Watchman/inotify support and other ways to speed up git status","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-11-02T20:56:16Z","receivedAt":"2015-11-02T20:56:16Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Thu, 2015-10-29 at 09:10 +0100, Christian Couder wrote:\n> > We're using Watchman at Twitter.  A week or two ago posted a dump of our\n> > code to github, but I would advise waiting a day or two to use it, as\n> > I'm about to pull a large number of bugfixes into it (I'll update this\n> > thread and provide a link once I do so).\n> \n> Great, I will have a look at it then!\n\nHere's the most recent version:\n\nhttps://github.com/dturner-tw/git/tree/dturner/watchman\n"},{"id":"272784","messageId":"CACsJy8AC1B+b_KBBcob07LkAkjjiwFQCLnMcgX0iUdQeNdeLfQ@mail.gmail.com","threadId":"40615","inReplyTo":"1446497776.4131.6.camel@twopensource.com","subject":"Re: Watchman/inotify support and other ways to speed up git status","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-11-03T05:45:22Z","receivedAt":"2015-11-03T05:45:22Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 2, 2015 at 9:56 PM, David Turner <dturner@twopensource.com> wrote:\n> On Thu, 2015-10-29 at 09:10 +0100, Christian Couder wrote:\n>> > We're using Watchman at Twitter.  A week or two ago posted a dump of our\n>> > code to github, but I would advise waiting a day or two to use it, as\n>> > I'm about to pull a large number of bugfixes into it (I'll update this\n>> > thread and provide a link once I do so).\n>>\n>> Great, I will have a look at it then!\n>\n> Here's the most recent version:\n>\n> https://github.com/dturner-tw/git/tree/dturner/watchman\n\nChristian, the index-helper/watchman series are posted because you\nshowed interest in this area. I'm not rerolling to address David's\ncomments on the series for now. Take your time evaluate the two\napproaches, then you can pick one (and let me know if you want me to\nhand my series over, very glad to do so).\n-- \nDuy\n"},{"id":"272788","messageId":"CAP8UFD04C-Amt2vGQKrJmA=DYVXrR7+hhkk9e49NsT-F0wiV1Q@mail.gmail.com","threadId":"40615","inReplyTo":"CACsJy8AC1B+b_KBBcob07LkAkjjiwFQCLnMcgX0iUdQeNdeLfQ@mail.gmail.com","subject":"Re: Watchman/inotify support and other ways to speed up git status","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2015-11-03T07:09:53Z","receivedAt":"2015-11-03T07:09:53Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Nov 3, 2015 at 6:45 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Mon, Nov 2, 2015 at 9:56 PM, David Turner <dturner@twopensource.com> wrote:\n>> On Thu, 2015-10-29 at 09:10 +0100, Christian Couder wrote:\n>>> > We're using Watchman at Twitter.  A week or two ago posted a dump of our\n>>> > code to github, but I would advise waiting a day or two to use it, as\n>>> > I'm about to pull a large number of bugfixes into it (I'll update this\n>>> > thread and provide a link once I do so).\n>>>\n>>> Great, I will have a look at it then!\n>>\n>> Here's the most recent version:\n>>\n>> https://github.com/dturner-tw/git/tree/dturner/watchman\n>\n> Christian, the index-helper/watchman series are posted because you\n> showed interest in this area. I'm not rerolling to address David's\n> comments on the series for now.\n\nOk no problem. Thanks a lot to you and David for posting your rebased series!\n\n> Take your time evaluate the two\n> approaches, then you can pick one (and let me know if you want me to\n> hand my series over, very glad to do so).\n\nYeah, I will do that, thanks again!\n"},{"id":"272848","messageId":"1446582741.4131.37.camel@twopensource.com","threadId":"40615","inReplyTo":"CAP8UFD04C-Amt2vGQKrJmA=DYVXrR7+hhkk9e49NsT-F0wiV1Q@mail.gmail.com","subject":"Re: Watchman/inotify support and other ways to speed up git status","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2015-11-03T20:32:21Z","receivedAt":"2015-11-03T20:32:21Z","isPatch":false,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2015-11-03 at 08:09 +0100, Christian Couder wrote:\n> On Tue, Nov 3, 2015 at 6:45 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n> > On Mon, Nov 2, 2015 at 9:56 PM, David Turner <dturner@twopensource.com> wrote:\n> >> On Thu, 2015-10-29 at 09:10 +0100, Christian Couder wrote:\n> >>> > We're using Watchman at Twitter.  A week or two ago posted a dump of our\n> >>> > code to github, but I would advise waiting a day or two to use it, as\n> >>> > I'm about to pull a large number of bugfixes into it (I'll update this\n> >>> > thread and provide a link once I do so).\n> >>>\n> >>> Great, I will have a look at it then!\n> >>\n> >> Here's the most recent version:\n> >>\n> >> https://github.com/dturner-tw/git/tree/dturner/watchman\n> >\n> > Christian, the index-helper/watchman series are posted because you\n> > showed interest in this area. I'm not rerolling to address David's\n> > comments on the series for now.\n> \n> Ok no problem. Thanks a lot to you and David for posting your rebased series!\n> \n> > Take your time evaluate the two\n> > approaches, then you can pick one (and let me know if you want me to\n> > hand my series over, very glad to do so).\n> \n> Yeah, I will do that, thanks again!\n\nTo be clear: I think Duy's approach is probably best in the long term.\n"}]}