{"thread":{"id":"38736","subject":"Git very slow ?","startedAt":"2015-03-07T01:30:07Z","lastAt":"2015-03-08T23:31:54Z","messageCount":9,"participants":["Ken Moffat","Kevin D","David Kastrup","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"257219","messageId":"20150307013007.GA13250@milliways","threadId":"38736","inReplyTo":null,"subject":"Git very slow ?","fromName":"Ken Moffat","fromEmail":"zarniwhoop@ntlworld.com","sentAt":"2015-03-07T01:30:07Z","receivedAt":"2015-03-07T01:30:07Z","isPatch":false,"sender":{"key":"zarniwhoop@ntlworld.com","avatar":null},"body":"Hi, please CC me if that is not your usual fashion, because I am not\nsubscribed.\n\nI use git for my build scripts - those are accessed over nfs.  Since\nI started using 2.1 and later (I don't think I used 2.0) commands\nsuch as 'commit' take a long time before anything happens.  I\nassumed that the newer version meant this would take longer.\n\nBut today^Wyesterday I was bisecting the kernel on a local\nfilesystem - even when the number of revisions left to test was in\nthe single digits, git bisect took a long time to decide which\nrevision should be the next one to test.  The filesystems are ext4.\nIs this sort of delay normal now?\n\nWhat really prompted me to ask is that I ran git blame on a script,\nto see when I made a particular change so that I could add that\ninformation to a ticket, and almost gave up waiting because it felt\nas if it was taking for ever.\n\nĸen\n-- \nNanny Ogg usually went to bed early. After all, she was an old lady.\nSometimes she went to bed as early as 6 a.m.\n"},{"id":"257330","messageId":"20150308155136.GA6273@vps892.directvps.nl","threadId":"38736","inReplyTo":"20150307013007.GA13250@milliways","subject":"Re: Git very slow ?","fromName":"Kevin D","fromEmail":"me@ikke.info","sentAt":"2015-03-08T15:51:36Z","receivedAt":"2015-03-08T15:51:36Z","isPatch":false,"sender":{"key":"me@ikke.info","avatar":"https://avatars.githubusercontent.com/u/135698?v=4"},"body":"On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:\n> Hi, please CC me if that is not your usual fashion, because I am not\n> subscribed.\n> \n> I use git for my build scripts - those are accessed over nfs.  Since\n> I started using 2.1 and later (I don't think I used 2.0) commands\n> such as 'commit' take a long time before anything happens.  I\n> assumed that the newer version meant this would take longer.\n> \n> But today^Wyesterday I was bisecting the kernel on a local\n> filesystem - even when the number of revisions left to test was in\n> the single digits, git bisect took a long time to decide which\n> revision should be the next one to test.  The filesystems are ext4.\n> Is this sort of delay normal now?\n> \n> What really prompted me to ask is that I ran git blame on a script,\n> to see when I made a particular change so that I could add that\n> information to a ticket, and almost gave up waiting because it felt\n> as if it was taking for ever.\n> \n\nWhat kind of repository are we talking about? How many files, how big?\nGit should not have become significantly slower recently.\n\nAlso, might there be anti-virus software that slows down file access?\n"},{"id":"257332","messageId":"87zj7nmpdp.fsf@fencepost.gnu.org","threadId":"38736","inReplyTo":"20150308155136.GA6273@vps892.directvps.nl","subject":"Re: Git very slow ?","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2015-03-08T16:21:22Z","receivedAt":"2015-03-08T16:21:22Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Kevin D <me@ikke.info> writes:\n\n> On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:\n>> Hi, please CC me if that is not your usual fashion, because I am not\n>> subscribed.\n>> \n>> I use git for my build scripts - those are accessed over nfs.  Since\n>> I started using 2.1 and later (I don't think I used 2.0) commands\n>> such as 'commit' take a long time before anything happens.  I\n>> assumed that the newer version meant this would take longer.\n>> \n>> But today^Wyesterday I was bisecting the kernel on a local\n>> filesystem - even when the number of revisions left to test was in\n>> the single digits, git bisect took a long time to decide which\n>> revision should be the next one to test.  The filesystems are ext4.\n>> Is this sort of delay normal now?\n>> \n>> What really prompted me to ask is that I ran git blame on a script,\n>> to see when I made a particular change so that I could add that\n>> information to a ticket, and almost gave up waiting because it felt\n>> as if it was taking for ever.\n>> \n>\n> What kind of repository are we talking about? How many files, how big?\n> Git should not have become significantly slower recently.\n\nParticularly not git-blame in 2.1.  I should be quite surprised to see\nany git-blame call running noticeably slower in 2.1 than in any\npreceding version.\n\nWhat may have happened is that the repository recently got repacked\naggressively and thus any access to older revisions got slower.\nHowever, that change would be mostly tied to the repository rather than\nthe version of Git you access it with.\n\n-- \nDavid Kastrup\n"},{"id":"257336","messageId":"20150308190244.GA32504@milliways","threadId":"38736","inReplyTo":"20150308155136.GA6273@vps892.directvps.nl","subject":"Re: Git very slow ?","fromName":"Ken Moffat","fromEmail":"zarniwhoop@ntlworld.com","sentAt":"2015-03-08T19:02:44Z","receivedAt":"2015-03-08T19:02:44Z","isPatch":false,"sender":{"key":"zarniwhoop@ntlworld.com","avatar":null},"body":"On Sun, Mar 08, 2015 at 04:51:36PM +0100, Kevin D wrote:\n> On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:\n> > Hi, please CC me if that is not your usual fashion, because I am not\n> > subscribed.\n> > \n> > I use git for my build scripts - those are accessed over nfs.  Since\n> > I started using 2.1 and later (I don't think I used 2.0) commands\n> > such as 'commit' take a long time before anything happens.  I\n> > assumed that the newer version meant this would take longer.\n> > \n> > But today^Wyesterday I was bisecting the kernel on a local\n> > filesystem - even when the number of revisions left to test was in\n> > the single digits, git bisect took a long time to decide which\n> > revision should be the next one to test.  The filesystems are ext4.\n> > Is this sort of delay normal now?\n> > \n> > What really prompted me to ask is that I ran git blame on a script,\n> > to see when I made a particular change so that I could add that\n> > information to a ticket, and almost gave up waiting because it felt\n> > as if it was taking for ever.\n> > \n> \n> What kind of repository are we talking about? How many files, how big?\n> Git should not have become significantly slower recently.\n> \n\nThe comments on git bisect were for linus'skernel tree, on a local\ndisk.  2.3GB of repo, just under 57000 files.\n\nMy own repo of build scripts, where I have noticed the delay before\ngit commit lets me type in the message, is an nfs v3 mount from\nanother of my machines in the same room - ping between them gives\ntimes of 0.25 to 0.3 seconds and I think the nfs part is irrelevant.\nHere, the size is 70MB and 12133 files [ about 1500 scripts total,\nso the rest is from the commits ].\n\nSome of this might be the drives - on the desktop with linus's tree\nthe machine only supports SATA2 (3GB/S), but the machine serving my\nscripts goes back further and probably only supports SATA1 (1.5GB/S)\n\n> Also, might there be anti-virus software that slows down file access?\n\nNo, this is all local access on linux machines.\n\nĸen\n-- \nNanny Ogg usually went to bed early. After all, she was an old lady.\nSometimes she went to bed as early as 6 a.m.\n"},{"id":"257338","messageId":"20150308192045.GB32504@milliways","threadId":"38736","inReplyTo":"87zj7nmpdp.fsf@fencepost.gnu.org","subject":"Re: Git very slow ?","fromName":"Ken Moffat","fromEmail":"zarniwhoop@ntlworld.com","sentAt":"2015-03-08T19:20:46Z","receivedAt":"2015-03-08T19:20:46Z","isPatch":false,"sender":{"key":"zarniwhoop@ntlworld.com","avatar":null},"body":"On Sun, Mar 08, 2015 at 05:21:22PM +0100, David Kastrup wrote:\n> Kevin D <me@ikke.info> writes:\n> \n> > On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:\n> >> Hi, please CC me if that is not your usual fashion, because I am not\n> >> subscribed.\n> >> \n> >> I use git for my build scripts - those are accessed over nfs.  Since\n> >> I started using 2.1 and later (I don't think I used 2.0) commands\n> >> such as 'commit' take a long time before anything happens.  I\n> >> assumed that the newer version meant this would take longer.\n> >> \n> >> But today^Wyesterday I was bisecting the kernel on a local\n> >> filesystem - even when the number of revisions left to test was in\n> >> the single digits, git bisect took a long time to decide which\n> >> revision should be the next one to test.  The filesystems are ext4.\n> >> Is this sort of delay normal now?\n> >> \n> >> What really prompted me to ask is that I ran git blame on a script,\n> >> to see when I made a particular change so that I could add that\n> >> information to a ticket, and almost gave up waiting because it felt\n> >> as if it was taking for ever.\n> >> \n> >\n> > What kind of repository are we talking about? How many files, how big?\n> > Git should not have become significantly slower recently.\n> \n> Particularly not git-blame in 2.1.  I should be quite surprised to see\n> any git-blame call running noticeably slower in 2.1 than in any\n> preceding version.\n> \n> What may have happened is that the repository recently got repacked\n> aggressively and thus any access to older revisions got slower.\n> However, that change would be mostly tied to the repository rather than\n> the version of Git you access it with.\n> \nThat is possible - well, not recently-recently, but I might have\nrepacked my repo of buildscripts some time last year.  Running\n ls -al .git\nin that repository gives me:\ndrwxr-xr-x   8 ken 100   4096 Mar  8 16:08 .\ndrwxr-xr-x  48 ken 100   4096 Mar  8 03:05 ..\n-rw-r--r--   1 ken 100    220 May 12  2014 BRANCH_DESCRIPTION\ndrwxr-xr-x   2 ken 100   4096 Apr 13  2010 branches\n-rw-r--r--   1 ken 100    470 Mar  8 16:08 COMMIT_EDITMSG\n-rw-r--r--   1 ken 100    566 May 17  2014 config\n-rw-r--r--   1 ken 100     73 May  1  2010 description\n-rw-r--r--   1 ken 100 196439 Sep 17 21:56 gitk.cache\n-rw-rw-rw-   1 ken 100     29 Feb  8 22:19 HEAD\ndrwxr-xr-x   2 ken 100   4096 May  1  2010 hooks\n-rw-r--r--   1 ken 100 218255 Mar  8 16:07 index\ndrwxr-xr-x   2 ken 100   4096 Sep 16  2013 info\ndrwxr-xr-x   3 ken 100   4096 Sep 16  2013 logs\ndrwxr-xr-x 260 ken 100   4096 Nov 12  2013 objects\n-rw-r--r--   1 ken 100     41 Nov 11 06:05 ORIG_HEAD\n-rw-r--r--   1 ken 100   1879 Sep 16  2013 packed-refs\ndrwxr-xr-x   5 ken 100   4096 May 20  2014 refs\n-rw-r--r--   1 ken 100     41 Dec  7  2010 RENAMED-REF\n\nRunning git blame on a script which dates back to when the repo was\ncreated takes between 5 and 6 seconds to show the first screen,\nrunning it on a file first created last August took about 3 seconds.\nI have another small repo also on nfs with only a few files, started\nlast year, and there git blame takes about a second.\n\nIn my clone of linus' tree, running git blame scripts/headers.sh (a\nrandom example,I do not normally run git blame there) took 10\nseconds to return the first screen.\n\nAll of those timings are from preparing the comand, watching my\ndesktop clock until it changes the second, and hitting enter - so,\nonly accurate to approx one second.\n\nĸen\n-- \nNanny Ogg usually went to bed early. After all, she was an old lady.\nSometimes she went to bed as early as 6 a.m.\n"},{"id":"257340","messageId":"87sidfmgag.fsf@fencepost.gnu.org","threadId":"38736","inReplyTo":"20150308192045.GB32504@milliways","subject":"Re: Git very slow ?","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2015-03-08T19:37:43Z","receivedAt":"2015-03-08T19:37:43Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Ken Moffat <zarniwhoop@ntlworld.com> writes:\n\n> On Sun, Mar 08, 2015 at 05:21:22PM +0100, David Kastrup wrote:\n>\n>> Particularly not git-blame in 2.1.  I should be quite surprised to see\n>> any git-blame call running noticeably slower in 2.1 than in any\n>> preceding version.\n>> \n>> What may have happened is that the repository recently got repacked\n>> aggressively and thus any access to older revisions got slower.\n>> However, that change would be mostly tied to the repository rather than\n>> the version of Git you access it with.\n>> \n> That is possible - well, not recently-recently, but I might have\n> repacked my repo of buildscripts some time last year.  Running\n>  ls -al .git\n> in that repository gives me:\n> drwxr-xr-x   8 ken 100   4096 Mar  8 16:08 .\n> drwxr-xr-x  48 ken 100   4096 Mar  8 03:05 ..\n> -rw-r--r--   1 ken 100    220 May 12  2014 BRANCH_DESCRIPTION\n> drwxr-xr-x   2 ken 100   4096 Apr 13  2010 branches\n> -rw-r--r--   1 ken 100    470 Mar  8 16:08 COMMIT_EDITMSG\n> -rw-r--r--   1 ken 100    566 May 17  2014 config\n> -rw-r--r--   1 ken 100     73 May  1  2010 description\n> -rw-r--r--   1 ken 100 196439 Sep 17 21:56 gitk.cache\n> -rw-rw-rw-   1 ken 100     29 Feb  8 22:19 HEAD\n> drwxr-xr-x   2 ken 100   4096 May  1  2010 hooks\n> -rw-r--r--   1 ken 100 218255 Mar  8 16:07 index\n> drwxr-xr-x   2 ken 100   4096 Sep 16  2013 info\n> drwxr-xr-x   3 ken 100   4096 Sep 16  2013 logs\n> drwxr-xr-x 260 ken 100   4096 Nov 12  2013 objects\n> -rw-r--r--   1 ken 100     41 Nov 11 06:05 ORIG_HEAD\n> -rw-r--r--   1 ken 100   1879 Sep 16  2013 packed-refs\n> drwxr-xr-x   5 ken 100   4096 May 20  2014 refs\n> -rw-r--r--   1 ken 100     41 Dec  7  2010 RENAMED-REF\n>\n> Running git blame on a script which dates back to when the repo was\n> created takes between 5 and 6 seconds to show the first screen,\n\nSince git blame outputs everything once it is finished (\"the first\nscreen\" is purely the pager's business), it needs to unpack the entire\nhistory of the file (unless no blameable lines remain at all) and look\nat it.  6 seconds tends not to be all that excessive for extracting more\nthan 5 years of a file's history.\n\n-- \nDavid Kastrup\n"},{"id":"257339","messageId":"CA+55aFwn9vOWAL_9eHQt+kQN16j3+nQLeibWos86g5T_RZEizg@mail.gmail.com","threadId":"38736","inReplyTo":"20150308190244.GA32504@milliways","subject":"Re: Git very slow ?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2015-03-08T19:39:10Z","receivedAt":"2015-03-08T19:39:10Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Sun, Mar 8, 2015 at 12:02 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:\n>\n> The comments on git bisect were for linus'skernel tree, on a local\n> disk.  2.3GB of repo, just under 57000 files.\n\nUgh. I hope you are talking about checked-out size.\n\n    [torvalds@i7 linux]$ du -sh .git\n   850M .git\n\nbecause otherwise it sounds like that repo hasn't been repacked in forever.\n\nTo really pack things (which can slow things down for old history as\npeople said, but on the whole it tends to be a big win due to less\nIO), do\n\n   git repack -adf --window=200 --depth=200\n\nand go away for a while. Oh, and make sure your machine has enough\nmemory and CPU to make that \"for a while\" not be *too* long.\n\nYou should have a few hundred files (just a few tens of files directly\nafter the repack) and that roughly 850MB of space for the repository\ninformation itself.\n\nBut yeah, fully checked out and built with all the modules etc, you\nwould have much more.\n\nThat said, if you have something fairly that is consistently really\nslow (like the \"git commit\" you mentioned), *before* doing the repack,\ndo\n\n   strace -o ../trace-file -Ttt git commit\n\nand we can get a much better guess about why it's so slow. Send it to\nme in private email if you don't want to make it public, and I can\ntake a look.\n\n> ping between them gives times of 0.25 to 0.3 seconds\n\n.. and I *really* hope that was not seconds, but ms. Otherwise your\nnfsv3 setup is going to be really really painful.\n\n                          Linus\n"},{"id":"257342","messageId":"CA+55aFzDRg4kHHGGHd91kVxfj8eX0g1w5T7SyN_CouCf=_tW3A@mail.gmail.com","threadId":"38736","inReplyTo":"87sidfmgag.fsf@fencepost.gnu.org","subject":"Re: Git very slow ?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2015-03-08T19:46:07Z","receivedAt":"2015-03-08T19:46:07Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"On Sun, Mar 8, 2015 at 12:37 PM, David Kastrup <dak@gnu.org> wrote:\n>\n> Since git blame outputs everything once it is finished (\"the first\n> screen\" is purely the pager's business), it needs to unpack the entire\n> history of the file (unless no blameable lines remain at all) and look\n> at it.  6 seconds tends not to be all that excessive for extracting more\n> than 5 years of a file's history.\n\nYeah, \"git blame\" can easily be several seconds without anything being wrong.\n\nBut \"git commit\" should be fairly instantaneous. Even over NFS.\n\nThat said, on NFS in particular, make sure you don't have\n\n    [core]\n        PreloadIndex = false\n\nin your .gitconfig to disable the threaded index preloading.\n\nBut \"core.preloadindex\" _should_ be enabled by default in anything but\nthe most ancient git versions, and it can make a huge difference on\nNFS because it allows the 'lstat()' calls to check that the index is\nup-to-date to be done in parallel. Without that, git on NFS can be a\nbit sluggish.\n\nOn local filesystems it normally doesn't make as much of a difference,\nsince things tend to be either cached or seek-limited.\n\n                             Linus\n"},{"id":"257353","messageId":"20150308233154.GB3792@milliways","threadId":"38736","inReplyTo":"CA+55aFwn9vOWAL_9eHQt+kQN16j3+nQLeibWos86g5T_RZEizg@mail.gmail.com","subject":"Re: Git very slow ?","fromName":"Ken Moffat","fromEmail":"zarniwhoop@ntlworld.com","sentAt":"2015-03-08T23:31:54Z","receivedAt":"2015-03-08T23:31:54Z","isPatch":false,"sender":{"key":"zarniwhoop@ntlworld.com","avatar":null},"body":"On Sun, Mar 08, 2015 at 12:39:10PM -0700, Linus Torvalds wrote:\n> On Sun, Mar 8, 2015 at 12:02 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:\n> >\n> > The comments on git bisect were for linus'skernel tree, on a local\n> > disk.  2.3GB of repo, just under 57000 files.\n> \n> Ugh. I hope you are talking about checked-out size.\n> \n>     [torvalds@i7 linux]$ du -sh .git\n>    850M .git\n> \n> because otherwise it sounds like that repo hasn't been repacked in forever.\n> \n\n Yes - I had finished bisecting, with the build still present.\n\n> To really pack things (which can slow things down for old history as\n> people said, but on the whole it tends to be a big win due to less\n> IO), do\n> \n>    git repack -adf --window=200 --depth=200\n> \n> and go away for a while. Oh, and make sure your machine has enough\n> memory and CPU to make that \"for a while\" not be *too* long.\n> \n\n For that, many thanks - this desktop has about 7GB (integrated\ngraphics steal a bit), current AMD desktop, and the repack of my\nscripts repo took about 56 seconds.  I'll do that on my copy of\nthe kernel tomorrow (it's on another machine).\n\n> You should have a few hundred files (just a few tens of files directly\n> after the repack) and that roughly 850MB of space for the repository\n> information itself.\n> \n> But yeah, fully checked out and built with all the modules etc, you\n> would have much more.\n> \n> That said, if you have something fairly that is consistently really\n> slow (like the \"git commit\" you mentioned), *before* doing the repack,\n> do\n> \n>    strace -o ../trace-file -Ttt git commit\n> \n> and we can get a much better guess about why it's so slow. Send it to\n> me in private email if you don't want to make it public, and I can\n> take a look.\n\n I don't think you need to look - it was taking most of the time\n(about 8 seconds) looping through the many files below .git/objects.\nThe trace was just over 9000 lines, repeating after the repack was\nless than 1300 lines.  It's available (97K after using xz) if you\nthink it would be useful, but I think you have already diagnosed the\nproblem and solution.\n> \n> > ping between them gives times of 0.25 to 0.3 seconds\n> \n> .. and I *really* hope that was not seconds, but ms. Otherwise your\n> nfsv3 setup is going to be really really painful.\n> \n>                           Linus\n\n Yes, I'm not always good at knowing the right units.  Thanks for the\nhelp.\n\nĸen\n-- \nNanny Ogg usually went to bed early. After all, she was an old lady.\nSometimes she went to bed as early as 6 a.m.\n"}]}