{"thread":{"id":"14150","subject":"update-index --assume-unchanged doesn't make things go fast","startedAt":"2008-06-25T16:44:30Z","lastAt":"2008-06-28T02:03:49Z","messageCount":16,"participants":["Avery Pennarun","Michael J Gruber","Jakub Narebski","Junio C Hamano","Stephen R. van den Berg","Dana How"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"81142","messageId":"32541b130806250944x717cf609x7aa520c77a7c6911@mail.gmail.com","threadId":"14150","inReplyTo":null,"subject":"update-index --assume-unchanged doesn't make things go fast","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-06-25T16:44:30Z","receivedAt":"2008-06-25T16:44:30Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"Hi all,\n\nUsing git 1.5.6.64.g85fe, but this applies to various other versions I've tried.\n\nI have a git repo with about 17000+ files in 1000+ directories.  In\nLinux, \"git status\" runs in under a second, which is perfectly fine.\nBut on Windows, which can apparently only stat() about 1000 files per\nsecond, \"git status\" takes at least 17 seconds to run, even with a hot\ncache.  (I've confirmed that stat() is so slow on Windows by writing a\nsimple program that just runs stat() in a tight loop.  The slowness\nmay be cygwin-related, as I found some direct Win32 calls that seem to\ngo more than twice as fast... which is still too slow.)\n\n\"git status\" is not so important, since I can choose not to run it.\nBut it turns out that every git checkout and git commit does all the\nsame stuff, which is really not so great.  Even worse if you consider\nthat \"git status\" is almost always what I do by hand anyway to check\nthings before I commit.\n\nSo anyway, I read about the git-update-index --assume-unchanged\noption, and thought that might be just what I want.  So I did this\n(back in Linux, where things are easier to debug):\n\n$ strace -fe lstat64 git status 2>&1 | wc -l\n17869\n\n$ git ls-files | xargs -d '\\n' git update-index --assume-unchanged\n\n$ strace -fe lstat64 git status 2>&1 | wc -l\n33\n\nSo far, so good, and \"git status\" is now noticeably faster on my Linux\nsystem (maybe twice as fast).  It's also noticeably faster on my\nWindows system, but not as fast as I would have hoped.  I've tracked\nit down to this:\n\n$ strace -fe getdents64 git status 2>&1 | wc -l\n2729\n\n\"git status\" still checks all the *directories* to see if there are\nany new files.  Of course!  --assume-unchanged can't be applied to a\ndirectory, so there's no way to tell it not to do so.\n\nAlso, \"git diff\" is still as slow as ever:\n\n$ strace -fe lstat64 git diff 2>&1 | wc -l\n23199\n\nIt seems to be stat()ing the files even though they are\n--assume-unchanged, which is probably a simple bug.\n\nAnd while we're here, \"git checkout\" seems to be working a lot harder\nthan it should be:\n\n$ strace -fe lstat64 git checkout -b boo 2>&1 | wc -l\n23227\n\nNote that I'm just creating a new branch name here, not even checking\nout any new files, so I can't think of any situation where the\ncheckout would fail.  Is there one?\n\nEven if I checkout a totally different branch, presumably it should\nonly need to stat() the files that changed between the old and new\nversions, right?  And that would normally be very fast.\n\nI don't mind doing some of the work to improve things here, as long as\npeople can give me some advice.  Specifically:\n\n1) What's a sensible way to tell git to *not* opendir() specific\ndirectories to look for unexpected files in \"git status\"?  (I don't\nthink I know enough to implement this myself.)\n\n2) Do you think git-diff should honour --assume-unchanged?  If not, why not?\n\n3) Do you think git-checkout can be optimized here?  I can see why it\nmight want to disregard --assume-unchanged (for safety reasons), but\npresumably it only needs to look at all at files that it's planning to\nchange, right?\n\n4) My idea is to eventually --assume-unchanged my whole repository,\nthen write a cheesy daemon that uses the Win32 dnotify-equivalent to\nwatch for files that get updated and then selectively\n--no-assume-unchanged files that it gets notified about.  That would\navoid the need to ever synchronously scan the whole repo for changes,\nthus making my git-Win32 experience much faster and more enjoyable.\n(This daemon ought to be possible to run on Linux as well, for similar\nimprovements on gigantic repositories.  Also note that TortoiseSVN for\nWindows does something similar to track file status updates, so this\nisn't *just* me being crazy.)\n\nThoughts?\n\nThanks,\n\nAvery\n"},{"id":"81146","messageId":"g3tvqd$2jj$1@ger.gmane.org","threadId":"14150","inReplyTo":"32541b130806250944x717cf609x7aa520c77a7c6911@mail.gmail.com","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Michael J Gruber","fromEmail":"michaeljgruber+gmane@fastmail.fm","sentAt":"2008-06-25T17:38:19Z","receivedAt":"2008-06-25T17:38:19Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Avery Pennarun venit, vidit, dixit 25.06.2008 18:44:\n...\n> 4) My idea is to eventually --assume-unchanged my whole repository,\n> then write a cheesy daemon that uses the Win32 dnotify-equivalent to\n> watch for files that get updated and then selectively\n> --no-assume-unchanged files that it gets notified about.  That would\n> avoid the need to ever synchronously scan the whole repo for changes,\n> thus making my git-Win32 experience much faster and more enjoyable.\n> (This daemon ought to be possible to run on Linux as well, for similar\n> improvements on gigantic repositories.  Also note that TortoiseSVN for\n> Windows does something similar to track file status updates, so this\n> isn't *just* me being crazy.)\n\nLooks like users on slow NFS would profit, too. Hate to say it, but hg \nfeels faster on (slow) NFS than git. Yet I use git, for other reasons ;)\n\nMichael\n"},{"id":"81152","messageId":"32541b130806251102l6e71a050o82fbd4f272d1d23f@mail.gmail.com","threadId":"14150","inReplyTo":"g3tvqd$2jj$1@ger.gmane.org","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-06-25T18:02:04Z","receivedAt":"2008-06-25T18:02:04Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 6/25/08, Michael J Gruber <michaeljgruber+gmane@fastmail.fm> wrote:\n> > 4) My idea is to eventually --assume-unchanged my whole repository,\n> > then write a cheesy daemon that uses the Win32 dnotify-equivalent to\n> > watch for files that get updated and then selectively\n> > --no-assume-unchanged files that it gets notified about.  That would\n> > avoid the need to ever synchronously scan the whole repo for changes,\n> > thus making my git-Win32 experience much faster and more enjoyable.\n> > (This daemon ought to be possible to run on Linux as well, for similar\n> > improvements on gigantic repositories.  Also note that TortoiseSVN for\n> > Windows does something similar to track file status updates, so this\n> > isn't *just* me being crazy.)\n>\n>  Looks like users on slow NFS would profit, too. Hate to say it, but hg\n> feels faster on (slow) NFS than git. Yet I use git, for other reasons ;)\n\nHmm, can you do dnotify over NFS?\n\nI'd like to know how hg goes any faster.  As far as I can see, git is\ngoing as fast as can be without some kind of daemon or other magic.\n(Except for my point #3, which seems relatively minor.)\n\nThanks,\n\nAvery\n"},{"id":"81166","messageId":"m33an1josg.fsf@localhost.localdomain","threadId":"14150","inReplyTo":"32541b130806250944x717cf609x7aa520c77a7c6911@mail.gmail.com","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-25T19:30:21Z","receivedAt":"2008-06-25T19:30:21Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Avery Pennarun\" <apenwarr@gmail.com> writes:\n\n> Hi all,\n> \n> Using git 1.5.6.64.g85fe, but this applies to various other versions\n> I've tried.\n> \n> I have a git repo with about 17000+ files in 1000+ directories.  In\n> Linux, \"git status\" runs in under a second, which is perfectly fine.\n> But on Windows, which can apparently only stat() about 1000 files per\n> second, \"git status\" takes at least 17 seconds to run, even with a hot\n> cache.  (I've confirmed that stat() is so slow on Windows by writing a\n> simple program that just runs stat() in a tight loop.  The slowness\n> may be cygwin-related, as I found some direct Win32 calls that seem to\n> go more than twice as fast... which is still too slow.)\n\nWhich git version do you use? Does it have the following configuration\nvariable (also available as command option):\n\n  status.showUntrackedFiles::\n        By default, linkgit:git-status[1] and linkgit:git-commit[1] show\n        files which are not currently tracked by Git. Directories which\n        contain only untracked files, are shown with the directory name\n        only. Showing untracked files means that Git needs to lstat() all\n        all the files in the whole repository, which might be slow on some\n        systems. So, this variable controls how the commands displays\n        the untracked files. Possible values are:\n\n        - 'no'     - Show no untracked files\n        - 'normal' - Shows untracked files and directories\n        - 'all'    - Shows also individual files in untracked directories.\n\nHTH.\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"81168","messageId":"7v1w2l1ew8.fsf@gitster.siamese.dyndns.org","threadId":"14150","inReplyTo":"m33an1josg.fsf@localhost.localdomain","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-06-25T19:41:11Z","receivedAt":"2008-06-25T19:41:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> \"Avery Pennarun\" <apenwarr@gmail.com> writes:\n>\n>> Hi all,\n>> \n>> Using git 1.5.6.64.g85fe, but this applies to various other versions\n>> I've tried.\n>> \n>> I have a git repo with about 17000+ files in 1000+ directories.  In\n>> Linux, \"git status\" runs in under a second, which is perfectly fine.\n>> But on Windows, which can apparently only stat() about 1000 files per\n>> second, \"git status\" takes at least 17 seconds to run, even with a hot\n>> cache.  (I've confirmed that stat() is so slow on Windows by writing a\n>> simple program that just runs stat() in a tight loop.  The slowness\n>> may be cygwin-related, as I found some direct Win32 calls that seem to\n>> go more than twice as fast... which is still too slow.)\n>\n> Which git version do you use? Does it have the following configuration\n> variable (also available as command option):\n>\n>   status.showUntrackedFiles::\n>         By default, linkgit:git-status[1] and linkgit:git-commit[1] show\n>         files which are not currently tracked by Git. Directories which\n>         contain only untracked files, are shown with the directory name\n>         only. Showing untracked files means that Git needs to lstat() all\n>         all the files in the whole repository, which might be slow on some\n>         systems. So, this variable controls how the commands displays\n>         the untracked files. Possible values are:\n>\n>         - 'no'     - Show no untracked files\n>         - 'normal' - Shows untracked files and directories\n>         - 'all'    - Shows also individual files in untracked directories.\n\nThat's on 'master' progressing forward to eventually become 1.6.0.\n"},{"id":"81171","messageId":"32541b130806251253t3dcada10nbf94fee9e4aed9ec@mail.gmail.com","threadId":"14150","inReplyTo":"m33an1josg.fsf@localhost.localdomain","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-06-25T19:53:33Z","receivedAt":"2008-06-25T19:53:33Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 6/25/08, Jakub Narebski <jnareb@gmail.com> wrote:\n> Which git version do you use? Does it have the following configuration\n>  variable (also available as command option):\n>\n>   status.showUntrackedFiles::\n> [...]\n\nThanks, I didn't know about that one.  Using that definitely makes\n\"git status\" go much faster (pretty much instantaneous if I've also\nused --assume-unchanged on everything).\n\nNow the catch is, if I want to implement the daemon I was talking\nabout earlier, I'd like to be able to notice untracked files (or\ndirectories with untracked files) individually.  Ideally, I guess the\nbest way would be to just keep a separate list of all existing files\nthat aren't in the index, and have git status look at that rather than\nat the actual filesystem.\n\nAre there any suggestions for how best to do this?\n\nThanks,\n\nAvery\n"},{"id":"81193","messageId":"200806252335.05083.jnareb@gmail.com","threadId":"14150","inReplyTo":"32541b130806251253t3dcada10nbf94fee9e4aed9ec@mail.gmail.com","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-25T21:35:03Z","receivedAt":"2008-06-25T21:35:03Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 25 Jun 2008, Avery Pennarun wrote:\n> On 6/25/08, Jakub Narebski <jnareb@gmail.com> wrote:\n> >\n> > Which git version do you use? Does it have the following configuration\n> > variable (also available as command option):\n> >\n> >   status.showUntrackedFiles::\n> > [...]\n> \n> Thanks, I didn't know about that one.  Using that definitely makes\n> \"git status\" go much faster (pretty much instantaneous if I've also\n> used --assume-unchanged on everything).\n> \n> Now the catch is, if I want to implement the daemon I was talking\n> about earlier, I'd like to be able to notice untracked files (or\n> directories with untracked files) individually.  Ideally, I guess the\n> best way would be to just keep a separate list of all existing files\n> that aren't in the index, and have git status look at that rather than\n> at the actual filesystem.\n> \n> Are there any suggestions for how best to do this?\n\nYou can try to take a look at how (third-party and Linux only) inotify\nextension for Mercurial works.  AFAIK IIRC it uses some kind of daemon\nwhich watches for inotify notices and updates Mercorial's equivalent\nof index.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"81222","messageId":"32541b130806251830t12b761bo9fa04a48cc53b2a9@mail.gmail.com","threadId":"14150","inReplyTo":"200806252335.05083.jnareb@gmail.com","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-06-26T01:30:34Z","receivedAt":"2008-06-26T01:30:34Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 6/25/08, Jakub Narebski <jnareb@gmail.com> wrote:\n> On Wed, 25 Jun 2008, Avery Pennarun wrote:\n>  > Now the catch is, if I want to implement the daemon I was talking\n>  > about earlier, I'd like to be able to notice untracked files (or\n>  > directories with untracked files) individually.  Ideally, I guess the\n>  > best way would be to just keep a separate list of all existing files\n>  > that aren't in the index, and have git status look at that rather than\n>  > at the actual filesystem.\n>  >\n>  > Are there any suggestions for how best to do this?\n>\n> You can try to take a look at how (third-party and Linux only) inotify\n>  extension for Mercurial works.  AFAIK IIRC it uses some kind of daemon\n>  which watches for inotify notices and updates Mercorial's equivalent\n>  of index.\n\nSorry, I asked the wrong question.  I wasn't asking how to implement\nthe daemon, which I think I can do without much trouble.  I actually\nneed to know how to represent the information.\n\nI was thinking of handling updated files by doing update-index\n--no-assume-unchanged on files that change.  But where should I store\ninformation about *untracked* files that have changed, so that\ngit-status can still report them but not have to scan them all?\n\nThanks,\n\nAvery\n"},{"id":"81256","messageId":"g3vl2l$qn7$1@ger.gmane.org","threadId":"14150","inReplyTo":"32541b130806251102l6e71a050o82fbd4f272d1d23f@mail.gmail.com","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Michael J Gruber","fromEmail":"michaeljgruber+gmane@fastmail.fm","sentAt":"2008-06-26T08:47:09Z","receivedAt":"2008-06-26T08:47:09Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Avery Pennarun venit, vidit, dixit 25.06.2008 20:02:\n> On 6/25/08, Michael J Gruber <michaeljgruber+gmane@fastmail.fm> wrote:\n>>> 4) My idea is to eventually --assume-unchanged my whole repository,\n>>> then write a cheesy daemon that uses the Win32 dnotify-equivalent to\n>>> watch for files that get updated and then selectively\n>>> --no-assume-unchanged files that it gets notified about.  That would\n>>> avoid the need to ever synchronously scan the whole repo for changes,\n>>> thus making my git-Win32 experience much faster and more enjoyable.\n>>> (This daemon ought to be possible to run on Linux as well, for similar\n>>> improvements on gigantic repositories.  Also note that TortoiseSVN for\n>>> Windows does something similar to track file status updates, so this\n>>> isn't *just* me being crazy.)\n>>  Looks like users on slow NFS would profit, too. Hate to say it, but hg\n>> feels faster on (slow) NFS than git. Yet I use git, for other reasons ;)\n> \n> Hmm, can you do dnotify over NFS?\n> \n> I'd like to know how hg goes any faster.  As far as I can see, git is\n> going as fast as can be without some kind of daemon or other magic.\n> (Except for my point #3, which seems relatively minor.)\n\nI haven't done any measurements, maybe I should; getting consistent \nresults would require setting up an isolated NFS environment, though.\n\nThe thing is that hg is very careful about serializing and minimizing \ndisk I/O, whereas git is very clever about delegating stuff to the \nkernel and processing data efficiently. In my work environment I have to \nkeep my repos on NFS. For heavy history rewriting I resort to /tmp or \n/dev/shm temporarily. But git status is kinda slow on NFS. I don't know \nabout [di]?notify over NFS.\n\nMichael\n"},{"id":"81271","messageId":"20080626112233.GA17625@cuci.nl","threadId":"14150","inReplyTo":"32541b130806250944x717cf609x7aa520c77a7c6911@mail.gmail.com","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Stephen R. van den Berg","fromEmail":"srb@cuci.nl","sentAt":"2008-06-26T11:22:33Z","receivedAt":"2008-06-26T11:22:33Z","isPatch":false,"sender":{"key":"srb@cuci.nl","avatar":"https://gravatar.com/avatar/f75389059e827634d38e9df2a9b6ecbd50028b5a454442efa1c7205b7ff29c6a?d=mp&s=160"},"body":"Avery Pennarun wrote:\n>1) What's a sensible way to tell git to *not* opendir() specific\n>directories to look for unexpected files in \"git status\"?  (I don't\n>think I know enough to implement this myself.)\n\nWould checking the mtime on the directory itself help?\n-- \nSincerely,\n           Stephen R. van den Berg.\n\nIf mind over matter is a matter of course, does it matter if nobody minds?\n"},{"id":"81436","messageId":"32541b130806271001t35eb97d2gb841e194b54f214@mail.gmail.com","threadId":"14150","inReplyTo":"20080626112233.GA17625@cuci.nl","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-06-27T17:01:20Z","receivedAt":"2008-06-27T17:01:20Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 6/26/08, Stephen R. van den Berg <srb@cuci.nl> wrote:\n> Avery Pennarun wrote:\n>  >1) What's a sensible way to tell git to *not* opendir() specific\n>  >directories to look for unexpected files in \"git status\"?  (I don't\n>  >think I know enough to implement this myself.)\n>\n> Would checking the mtime on the directory itself help?\n\nI'm guessing it would help somewhat (although not as much as not\nchecking anything at all).  However, we'd still have to check the\nmtime *against* something, and I don't think the index stores\ninformation about directories themselves.\n\nThanks,\n\nAvery\n"},{"id":"81445","messageId":"m3lk0qiy2i.fsf@localhost.localdomain","threadId":"14150","inReplyTo":"32541b130806271001t35eb97d2gb841e194b54f214@mail.gmail.com","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-27T17:31:53Z","receivedAt":"2008-06-27T17:31:53Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Avery Pennarun\" <apenwarr@gmail.com> writes:\n> On 6/26/08, Stephen R. van den Berg <srb@cuci.nl> wrote:\n>> Avery Pennarun wrote:\n>>>\n>>> 1) What's a sensible way to tell git to *not* opendir() specific\n>>> directories to look for unexpected files in \"git status\"?  (I don't\n>>> think I know enough to implement this myself.)\n>>\n>> Would checking the mtime on the directory itself help?\n> \n> I'm guessing it would help somewhat (although not as much as not\n> checking anything at all).  However, we'd still have to check the\n> mtime *against* something, and I don't think the index stores\n> information about directories themselves.\n\nBy the way, from time to time there on this mailing list is idea\nto add entries for directories in the index.  This could help situation\nlike yours, tracking emty directories, faster operations when some trees\nare unchanged, subtree <-> subproject changes.\n\nBut it always comes back to: 1.) no proposed implementation, 2.) \"git\ntracks contents\"...\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"81451","messageId":"32541b130806271056k4698a607r11e9fbaf9102e6f1@mail.gmail.com","threadId":"14150","inReplyTo":"m3lk0qiy2i.fsf@localhost.localdomain","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-06-27T17:56:22Z","receivedAt":"2008-06-27T17:56:22Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 6/27/08, Jakub Narebski <jnareb@gmail.com> wrote:\n> \"Avery Pennarun\" <apenwarr@gmail.com> writes:\n> > On 6/26/08, Stephen R. van den Berg <srb@cuci.nl> wrote:\n>  >> Avery Pennarun wrote:\n>  >>> 1) What's a sensible way to tell git to *not* opendir() specific\n>  >>> directories to look for unexpected files in \"git status\"?  (I don't\n>  >>> think I know enough to implement this myself.)\n>  >>\n>  >> Would checking the mtime on the directory itself help?\n>  >\n>  > I'm guessing it would help somewhat (although not as much as not\n>  > checking anything at all).  However, we'd still have to check the\n>  > mtime *against* something, and I don't think the index stores\n>  > information about directories themselves.\n>\n> By the way, from time to time there on this mailing list is idea\n>  to add entries for directories in the index.  This could help situation\n>  like yours, tracking emty directories, faster operations when some trees\n>  are unchanged, subtree <-> subproject changes.\n>\n>  But it always comes back to: 1.) no proposed implementation, 2.) \"git\n>  tracks contents\"...\n\nYes, I've seen the occasional discussions about this.\n\nI might volunteer to help solve (1) except that I have a feeling that\nchanging the index format would mangle all sorts of things beyond my\ncurrent understanding.  Attaining that understanding might not be so\nbad, except for (2), which seems like any proposed changes will\nprobably be rejected anyhow.\n\nSo naturally I was hoping for a magical alternative suggestion for my\ncurrent problem instead :)  One option I'm thinking about is to have\nmy proposed daemon keep its own \"index\", which tracks *all* the files\non the filesystem, not just the ones that have been\ngit-update-index'd.  Then anything that needs to compare against the\nfilesystem can choose to compare against the contents of this file\ninstead if it exists (and/or the right option is set, etc).  Does that\nsound sane?\n\nHave fun,\n\nAvery\n"},{"id":"81454","messageId":"56b7f5510806271109p58b4ce47ucdcd382faa463015@mail.gmail.com","threadId":"14150","inReplyTo":"32541b130806271056k4698a607r11e9fbaf9102e6f1@mail.gmail.com","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2008-06-27T18:09:56Z","receivedAt":"2008-06-27T18:09:56Z","isPatch":false,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On Fri, Jun 27, 2008 at 10:56 AM, Avery Pennarun <apenwarr@gmail.com> wrote:\n> On 6/27/08, Jakub Narebski <jnareb@gmail.com> wrote:\n>> \"Avery Pennarun\" <apenwarr@gmail.com> writes:\n>> > On 6/26/08, Stephen R. van den Berg <srb@cuci.nl> wrote:\n>>  >> Avery Pennarun wrote:\n>>  >>> 1) What's a sensible way to tell git to *not* opendir() specific\n>>  >>> directories to look for unexpected files in \"git status\"?  (I don't\n>>  >>> think I know enough to implement this myself.)\n>>  >>\n>>  >> Would checking the mtime on the directory itself help?\n>>  >\n>>  > I'm guessing it would help somewhat (although not as much as not\n>>  > checking anything at all).  However, we'd still have to check the\n>>  > mtime *against* something, and I don't think the index stores\n>>  > information about directories themselves.\n>>\n>> By the way, from time to time there on this mailing list is idea\n>>  to add entries for directories in the index.  This could help situation\n>>  like yours, tracking emty directories, faster operations when some trees\n>>  are unchanged, subtree <-> subproject changes.\n>>\n>>  But it always comes back to: 1.) no proposed implementation, 2.) \"git\n>>  tracks contents\"...\n>\n> Yes, I've seen the occasional discussions about this.\n>\n> I might volunteer to help solve (1) except that I have a feeling that\n> changing the index format would mangle all sorts of things beyond my\n> current understanding.  Attaining that understanding might not be so\n> bad, except for (2), which seems like any proposed changes will\n> probably be rejected anyhow.\n>\n> So naturally I was hoping for a magical alternative suggestion for my\n> current problem instead :)  One option I'm thinking about is to have\n> my proposed daemon keep its own \"index\", which tracks *all* the files\n> on the filesystem, not just the ones that have been\n> git-update-index'd.  Then anything that needs to compare against the\n> filesystem can choose to compare against the contents of this file\n> instead if it exists (and/or the right option is set, etc).  Does that\n> sound sane?\nIt sounds sane to me b/c I had the same reaction to this discussion.\nYou mean \"all the files in the _worktree_\" ?\nYou would use e.g. inotify on all the directories except .git?\nThis would be very helpful with an extremely large number of files.\n\nThanks,\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"81462","messageId":"32541b130806271151sdf6ce8hfeaa6d61515e8b5e@mail.gmail.com","threadId":"14150","inReplyTo":"56b7f5510806271109p58b4ce47ucdcd382faa463015@mail.gmail.com","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-06-27T18:51:38Z","receivedAt":"2008-06-27T18:51:38Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 6/27/08, Dana How <danahow@gmail.com> wrote:\n> It sounds sane to me b/c I had the same reaction to this discussion.\n>  You mean \"all the files in the _worktree_\" ?\n>  You would use e.g. inotify on all the directories except .git?\n>  This would be very helpful with an extremely large number of files.\n\nYes, that's the idea.  In Win32, it would use something other than\ninotify, but otherwise it should work about the same.\n\nAvery\n"},{"id":"81529","messageId":"7vd4m2l3i2.fsf@gitster.siamese.dyndns.org","threadId":"14150","inReplyTo":"m3lk0qiy2i.fsf@localhost.localdomain","subject":"Re: update-index --assume-unchanged doesn't make things go fast","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-06-28T02:03:49Z","receivedAt":"2008-06-28T02:03:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> writes:\n\n> By the way, from time to time there on this mailing list is idea\n> to add entries for directories in the index.  This could help situation\n> like yours, tracking emty directories, faster operations when some trees\n> are unchanged, subtree <-> subproject changes.\n\nTracking empty directories might be helped by having an explicit entry in\nthe index (even though it may not be the only possible implementation).  I\nhowever suspect you are overvaluing it for \"some trees are unchanged\"\ncase:\n\n        $ mkdir -p a/b\n        $ stat a | grep Modify\n        Modify: 2008-06-27 11:38:13.000000000 -0700\n        $ >a/b/c\n        $ stat a | grep Modify\n        Modify: 2008-06-27 11:38:13.000000000 -0700\n        $ >a/d\n        $ stat a | grep Modify\n        Modify: 2008-06-27 11:38:32.000000000 -0700\n\nYou have to descend into the leaf level anyway and directory mtime does\nnot allow you to check that much.\n"}]}