{"thread":{"id":"33304","subject":"git status takes 30 seconds on Windows 7. Why?","startedAt":"2013-03-27T16:39:31Z","lastAt":"2013-03-28T01:19:29Z","messageCount":12,"participants":["Jim Kinsman","Andreas Ericsson","Konstantin Khomoutov","Matthieu Moy","John Keeping","Jeff King","Linus Torvalds","Junio C Hamano","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"212400","messageId":"CAFT1WNyyXbCOQWMt8nYife7Tix_mJzDQrj5e+N=GKbDR8OyYzQ@mail.gmail.com","threadId":"33304","inReplyTo":null,"subject":"git status takes 30 seconds on Windows 7. Why?","fromName":"Jim Kinsman","fromEmail":"jakinsman@gmail.com","sentAt":"2013-03-27T16:39:31Z","receivedAt":"2013-03-27T16:39:31Z","isPatch":false,"sender":{"key":"jakinsman@gmail.com","avatar":null},"body":"git status takes 30 seconds on Windows 7. Here are some stats:\ngit ls-files | wc -l\n27330\n\ngit ls-files -o | wc -l\n4\n\n$ git diff --name-only | xargs du -chs\n68K update_import_contacts.php\n68K total\n\nWhat can I do??? This is so slow it is unbearable.\nBy the way i've done git gc several times and nothing changed.\n"},{"id":"212401","messageId":"51532203.4030303@op5.se","threadId":"33304","inReplyTo":"CAFT1WNyyXbCOQWMt8nYife7Tix_mJzDQrj5e+N=GKbDR8OyYzQ@mail.gmail.com","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2013-03-27T16:44:51Z","receivedAt":"2013-03-27T16:44:51Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"On 03/27/2013 05:39 PM, Jim Kinsman wrote:\n> git status takes 30 seconds on Windows 7. Here are some stats:\n> git ls-files | wc -l\n> 27330\n> \n> git ls-files -o | wc -l\n> 4\n> \n> $ git diff --name-only | xargs du -chs\n> 68K update_import_contacts.php\n> 68K total\n> \n> What can I do??? This is so slow it is unbearable.\n> By the way i've done git gc several times and nothing changed.\n\nI'm guessing it's the disk that's so slow. I accidentally put a git\nrepo on a network-mounted drive once. With 20ms round-trip time to\nthe server, git operations took forever.\n\nCould you try it on a disk you know is local? Preferrably a solid\nstate drive. If it's still slow there, we know for sure something's\nbroken inside git. If switching media causes git to become fast,\nyou'll know it's a hardware problem.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"212402","messageId":"20130327210207.be75b77c5cddd1e5c10764e0@domain007.com","threadId":"33304","inReplyTo":"CAFT1WNyyXbCOQWMt8nYife7Tix_mJzDQrj5e+N=GKbDR8OyYzQ@mail.gmail.com","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"Konstantin Khomoutov","fromEmail":"kostix+git@007spb.ru","sentAt":"2013-03-27T17:02:07Z","receivedAt":"2013-03-27T17:02:07Z","isPatch":false,"sender":{"key":"kostix+git@007spb.ru","avatar":null},"body":"On Wed, 27 Mar 2013 11:39:31 -0500\nJim Kinsman wrote:\n\n> git status takes 30 seconds on Windows 7. Here are some stats:\n[...]\n> What can I do??? This is so slow it is unbearable.\n> By the way i've done git gc several times and nothing changed.\n\nYou could try some voodoo [1] or experimental caching features [2].\n\n1. http://groups.google.com/group/msysgit/browse_thread/thread/02e3c0e046f07215\n2. http://groups.google.com/group/msysgit/browse_thread/thread/7cbfe3ca452650d1/93ce48e3875f7416\n"},{"id":"212411","messageId":"vpqr4j0rccf.fsf@grenoble-inp.fr","threadId":"33304","inReplyTo":"CAFT1WNyyXbCOQWMt8nYife7Tix_mJzDQrj5e+N=GKbDR8OyYzQ@mail.gmail.com","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2013-03-27T17:17:52Z","receivedAt":"2013-03-27T17:17:52Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jim Kinsman writes:\n\n> git status takes 30 seconds on Windows 7.\n\nAny anti-virus installed? They can interfer badly with disk-intensive\ntasks ...\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"212415","messageId":"20130327172216.GO2286@serenity.lan","threadId":"33304","inReplyTo":"CAFT1WNyyXbCOQWMt8nYife7Tix_mJzDQrj5e+N=GKbDR8OyYzQ@mail.gmail.com","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-03-27T17:22:17Z","receivedAt":"2013-03-27T17:22:17Z","isPatch":false,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Wed, Mar 27, 2013 at 11:39:31AM -0500, Jim Kinsman wrote:\n> git status takes 30 seconds on Windows 7. Here are some stats:\n> git ls-files | wc -l\n> 27330\n> \n> git ls-files -o | wc -l\n> 4\n> \n> $ git diff --name-only | xargs du -chs\n> 68K update_import_contacts.php\n> 68K total\n> \n> What can I do??? This is so slow it is unbearable.\n> By the way i've done git gc several times and nothing changed.\n\nCan you run these commands under \"time\" so that we can see that it's\ndefinitely the \"git ls-files\" taking 30 seconds and not something in\n$PS1?\n"},{"id":"212422","messageId":"CAFT1WNxm0Kx9vHx+NB92_tZV6KsVhefiVXa-K69YbWOkpCXSgA@mail.gmail.com","threadId":"33304","inReplyTo":"vpqr4j0rccf.fsf@grenoble-inp.fr","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"Jim Kinsman","fromEmail":"jakinsman@gmail.com","sentAt":"2013-03-27T18:15:43Z","receivedAt":"2013-03-27T18:15:43Z","isPatch":false,"sender":{"key":"jakinsman@gmail.com","avatar":null},"body":"The only anti-virus I have installed is Microsoft Security Essentials\nI turned off and it was still the same:\n$ cat /usr/bin/gitstatus\nstart_time=`date +%s`\ngit status && echo run time is $(expr `date +%s` - $start_time) s\n\n\n$ gitstatus\n# On branch test\n# Changes not staged for commit:\n# (use \"git add ...\" to update what will be committed)\n# (use \"git checkout -- ...\" to discard changes in working directory)\n#\n# modified: orgoptions.php\n# modified: update_import_contacts.php\n#\nno changes added to commit (use \"git add\" and/or \"git commit -a\")\nrun time is 10 s\n\nOn Wed, Mar 27, 2013 at 12:17 PM, Matthieu Moy\n wrote:\n> Jim Kinsman writes:\n>\n>> git status takes 30 seconds on Windows 7.\n>\n> Any anti-virus installed? They can interfer badly with disk-intensive\n> tasks ...\n>\n> --\n> Matthieu Moy\n> http://www-verimag.imag.fr/~moy/\n"},{"id":"212427","messageId":"20130327184657.GP2286@serenity.lan","threadId":"33304","inReplyTo":"CAFT1WNxm0Kx9vHx+NB92_tZV6KsVhefiVXa-K69YbWOkpCXSgA@mail.gmail.com","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-03-27T18:46:57Z","receivedAt":"2013-03-27T18:46:57Z","isPatch":false,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Wed, Mar 27, 2013 at 01:15:43PM -0500, Jim Kinsman wrote:\n> The only anti-virus I have installed is Microsoft Security Essentials\n> I turned off and it was still the same:\n> $ cat /usr/bin/gitstatus\n> start_time=`date +%s`\n> git status && echo run time is $(expr `date +%s` - $start_time) s\n> \n> \n> $ gitstatus\n> # On branch test\n> # Changes not staged for commit:\n> # (use \"git add ...\" to update what will be committed)\n> # (use \"git checkout -- ...\" to discard changes in working directory)\n> #\n> # modified: orgoptions.php\n> # modified: update_import_contacts.php\n> #\n> no changes added to commit (use \"git add\" and/or \"git commit -a\")\n> run time is 10 s\n\nThat doesn't seem hugely surprising to me. I have a moderately sized\nrepository (3047 files, although it's Java so there are some deep trees)\nand I get the following (Vista on a reasonably old laptop, best of 3,\nGit version 1.8.1.msysgit.1):\n\n$ time git ls-files >/dev/null\n\nreal\t0m0.047s\nuser\t0m0.015s\nsys\t0m0.015s\n\n$ time git status >/dev/null\n\nreal\t0m2.715s\nuser\t0m0.000s\nsys\t0m0.031s\n\n\nI'm not sure the \"user\" and \"sys\" times are correct, but the \"real\"\ntimes feel right. By comparison, on Linux on a much newer machine (so\nnot much of a comparison) on the same repository:\n\n$ time git status >/dev/null\n\nreal\t0m0.347s\nuser\t0m0.171s\nsys\t0m0.167s\n\n\nI think the simple reality is that Git was written with the assumption\nthat stat is cheap and that isn't really the case on Windows, where the\nfilesystem cache doesn't seem to do that well with this. It may be that\nGit's Windows compatibility code could do be made more efficient but I\nknow nothing about that, although a quick look in compat/mingw.c\nindicates that Git does already use its own stat implementations in\nplace of the MSys ones in search of speed.\n\n\nJohn\n"},{"id":"212432","messageId":"20130327190425.GA26380@sigill.intra.peff.net","threadId":"33304","inReplyTo":"20130327184657.GP2286@serenity.lan","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-03-27T19:04:25Z","receivedAt":"2013-03-27T19:04:25Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Mar 27, 2013 at 06:46:57PM +0000, John Keeping wrote:\n\n> I think the simple reality is that Git was written with the assumption\n> that stat is cheap and that isn't really the case on Windows, where the\n> filesystem cache doesn't seem to do that well with this.\n\nYes, I think that's pretty much the case (though most of my\nGit-on-Windows experience is from cygwin long ago, where the stat\nperformance was truly horrendous). Have you tried setting\ncore.preloadindex, which should run the stats in parallel?\n\n-Peff\n"},{"id":"212436","messageId":"CA+55aFypcwbLwPLq++AU9FggCKLYkgkuN6i-gOD9pRioH1Dz2g@mail.gmail.com","threadId":"33304","inReplyTo":"20130327190425.GA26380@sigill.intra.peff.net","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2013-03-27T19:27:23Z","receivedAt":"2013-03-27T19:27:23Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Wed, Mar 27, 2013 at 12:04 PM, Jeff King wrote:\n>\n> Yes, I think that's pretty much the case (though most of my\n> Git-on-Windows experience is from cygwin long ago, where the stat\n> performance was truly horrendous). Have you tried setting\n> core.preloadindex, which should run the stats in parallel?\n\nI wonder if preloadindex shouldn't be enabled by default.. It's a huge\ndeal on NFS, and the only real downside is that it expects threading\nto work. It potentially slows things down a tiny bit for single-CPU\ncases with everything cached, but that isn't likely to be a relevant\ncase.\n\nOf course, it can trigger filesystem scalability issues, and as a\nresult it will often not help very much if you have the bulk of your\nfiles in one (or a few) directories. But anybody who has so many files\nthat performance is an issue is not likely to have them all in one\nplace.\n\nAnd apparently the Windows FS metadata caching sucks, and things fall\nout of the cache for large trees. Color me not-very-surprised. It's\nprobably some size limit on the metadata that you can tweak. So I';m\nsure there's some registry setting or other that would make windows\nable to cache more than a few thousand filenames, and it would\nprobably improve performance a lot, but I do think preloadindex has\nbeen around long enough that it could just be the default.\n\nOf course, Jim should verify that preloadindex actually does solve his\nproblem. With 20k+ files, it should max out the 20 IO threads for\npreloading, and assuming the filesystem IO scales reasonably well, it\nshould fix the problem. But we do do a number of metadata ops\nsynchronously even with preloadindex, so things won't scale perfectly.\n\n(In particular: do open each directory and do the readdir stuff and\ntry to open .gitignore whether it exists or not. So you'll get\nsynchronous IO for each directory, but at least the per-file IO to\ncheck all the file stat data should scale).\n\n Linus\n"},{"id":"212440","messageId":"7vy5d8lijp.fsf@alter.siamese.dyndns.org","threadId":"33304","inReplyTo":"CA+55aFypcwbLwPLq++AU9FggCKLYkgkuN6i-gOD9pRioH1Dz2g@mail.gmail.com","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-27T20:00:26Z","receivedAt":"2013-03-27T20:00:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> On Wed, Mar 27, 2013 at 12:04 PM, Jeff King wrote:\n>>\n>> Yes, I think that's pretty much the case (though most of my\n>> Git-on-Windows experience is from cygwin long ago, where the stat\n>> performance was truly horrendous). Have you tried setting\n>> core.preloadindex, which should run the stats in parallel?\n>\n> I wonder if preloadindex shouldn't be enabled by default.\n\nI am surprised that we haven't done so.\n\nGiven that we haven't tweaked the parallelism or thread-cost\nparameters since the inception of the mechanism in Nov 2008, I\nsuspect that we would see praises from some and grievances from\nother corners of the user base for a while until we find acceptable\nvalues for them, but I agree the feature has been in use\nsufficiently by some people (heh, I just discovered that I don't\nhave it in my config), it can be the default.\n"},{"id":"212448","messageId":"CA+55aFzu72fxhms_azREiLz+hLDzT8XKJfJuYHZc=AQcZxmknw@mail.gmail.com","threadId":"33304","inReplyTo":"7vy5d8lijp.fsf@alter.siamese.dyndns.org","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2013-03-27T20:12:26Z","receivedAt":"2013-03-27T20:12:26Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Wed, Mar 27, 2013 at 1:00 PM, Junio C Hamano wrote:\n>\n> Given that we haven't tweaked the parallelism or thread-cost\n> parameters since the inception of the mechanism in Nov 2008, I\n> suspect that we would see praises from some and grievances from\n> other corners of the user base for a while until we find acceptable\n> values for them\n\nLooking at the parameters again, I really think they are pretty sane,\nand I don't think the numbers are all that likely to have shifted from\n2008. The maximum thread value is quite reasonable: twenty threads is\nsufficient to cover quite a bit of latency, and brings \"several\nseconds\" down to \"under half a second\" for any truly IO-limited load,\nwhile not being disastrous for the case where everything is in cache\nand we only have a limited number of CPU cores.\n\nAnd the \"at least 500 files per thread\" limit is eminently reasonable\ntoo - smaller projects like git won't have more than five or so\nthreads.\n\nSo I'd be very surprised if the values need much tweaking. Sure, there\nmight be some extreme cases that might tune for some particular\npatterns, and maybe we should make the values be tunable rather than\ntotally hardcoded, but I suspect there's limited up-side.\n\nIt might be interesting for the people who really like tuning, though.\nSo in addition to \"index.preload=true\", maybe an extended config\nformat like \"index_preload=50,200\" to say \"maximum of fifty threads,\nfor every 200 files\" could be done just so people could play around\nwith the numbers and see how much (if at all) they actually matter.\n\nBut I really don't think the original 20/500 rule is likely to be all\nthat bad for anybody. Unless there is some *really* sucky thread\nlibrary out there (ie fully user-space threads, so filename lookup\nisn't actually parallelised at all), but at least for that case the\nfix is to just say \"ok, your threads aren't real threads, so just\ndisable index preloading entirely).\n\n Linus\n"},{"id":"212487","messageId":"CACsJy8BgLpTPPzNPo9EQiUQq_s9Lru=ZsS+1B0CmU=5XjcHs-g@mail.gmail.com","threadId":"33304","inReplyTo":"CAFT1WNyyXbCOQWMt8nYife7Tix_mJzDQrj5e+N=GKbDR8OyYzQ@mail.gmail.com","subject":"Re: git status takes 30 seconds on Windows 7. Why?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-03-28T01:19:29Z","receivedAt":"2013-03-28T01:19:29Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Mar 27, 2013 at 11:39 PM, Jim Kinsman wrote:\n> git status takes 30 seconds on Windows 7. Here are some stats:\n> git ls-files | wc -l\n> 27330\n>\n> git ls-files -o | wc -l\n> 4\n>\n> $ git diff --name-only | xargs du -chs\n> 68K update_import_contacts.php\n> 68K total\n>\n> What can I do??? This is so slow it is unbearable.\n> By the way i've done git gc several times and nothing changed.\n\nYou can try \"status -uno\" to skip showing untracked files (and may be\ndo without -uno before commit so you don't miss files). You may also\ntry core.ignoreStat (but I think it's not very convenient to use)\n-- \nDuy\n"}]}