# Git very slow ? 9 messages from 2015-03-07 to 2015-03-08. Participants: Ken Moffat, Kevin D, David Kastrup, Linus Torvalds. Thread: https://gitlist.dev/t/38736 ## Ken Moffat, 2015-03-07 01:30 Subject: Git very slow ? Message-ID: <20150307013007.GA13250@milliways> URL: https://gitlist.dev/e/20150307013007.GA13250%40milliways ``` Hi, please CC me if that is not your usual fashion, because I am not subscribed. I use git for my build scripts - those are accessed over nfs. Since I started using 2.1 and later (I don't think I used 2.0) commands such as 'commit' take a long time before anything happens. I assumed that the newer version meant this would take longer. But today^Wyesterday I was bisecting the kernel on a local filesystem - even when the number of revisions left to test was in the single digits, git bisect took a long time to decide which revision should be the next one to test. The filesystems are ext4. Is this sort of delay normal now? What really prompted me to ask is that I ran git blame on a script, to see when I made a particular change so that I could add that information to a ticket, and almost gave up waiting because it felt as if it was taking for ever. ĸen -- Nanny Ogg usually went to bed early. After all, she was an old lady. Sometimes she went to bed as early as 6 a.m. ``` ## Kevin D, 2015-03-08 15:51 Subject: Re: Git very slow ? Message-ID: <20150308155136.GA6273@vps892.directvps.nl> URL: https://gitlist.dev/e/20150308155136.GA6273%40vps892.directvps.nl In-Reply-To: <20150307013007.GA13250@milliways> ``` On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote: > Hi, please CC me if that is not your usual fashion, because I am not > subscribed. > > I use git for my build scripts - those are accessed over nfs. Since > I started using 2.1 and later (I don't think I used 2.0) commands > such as 'commit' take a long time before anything happens. I > assumed that the newer version meant this would take longer. > > But today^Wyesterday I was bisecting the kernel on a local > filesystem - even when the number of revisions left to test was in > the single digits, git bisect took a long time to decide which > revision should be the next one to test. The filesystems are ext4. > Is this sort of delay normal now? > > What really prompted me to ask is that I ran git blame on a script, > to see when I made a particular change so that I could add that > information to a ticket, and almost gave up waiting because it felt > as if it was taking for ever. > What kind of repository are we talking about? How many files, how big? Git should not have become significantly slower recently. Also, might there be anti-virus software that slows down file access? ``` ## David Kastrup, 2015-03-08 16:21 Subject: Re: Git very slow ? Message-ID: <87zj7nmpdp.fsf@fencepost.gnu.org> URL: https://gitlist.dev/e/87zj7nmpdp.fsf%40fencepost.gnu.org In-Reply-To: <20150308155136.GA6273@vps892.directvps.nl> ``` Kevin D writes: > On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote: >> Hi, please CC me if that is not your usual fashion, because I am not >> subscribed. >> >> I use git for my build scripts - those are accessed over nfs. Since >> I started using 2.1 and later (I don't think I used 2.0) commands >> such as 'commit' take a long time before anything happens. I >> assumed that the newer version meant this would take longer. >> >> But today^Wyesterday I was bisecting the kernel on a local >> filesystem - even when the number of revisions left to test was in >> the single digits, git bisect took a long time to decide which >> revision should be the next one to test. The filesystems are ext4. >> Is this sort of delay normal now? >> >> What really prompted me to ask is that I ran git blame on a script, >> to see when I made a particular change so that I could add that >> information to a ticket, and almost gave up waiting because it felt >> as if it was taking for ever. >> > > What kind of repository are we talking about? How many files, how big? > Git should not have become significantly slower recently. Particularly not git-blame in 2.1. I should be quite surprised to see any git-blame call running noticeably slower in 2.1 than in any preceding version. What may have happened is that the repository recently got repacked aggressively and thus any access to older revisions got slower. However, that change would be mostly tied to the repository rather than the version of Git you access it with. -- David Kastrup ``` ## Ken Moffat, 2015-03-08 19:02 Subject: Re: Git very slow ? Message-ID: <20150308190244.GA32504@milliways> URL: https://gitlist.dev/e/20150308190244.GA32504%40milliways In-Reply-To: <20150308155136.GA6273@vps892.directvps.nl> ``` On Sun, Mar 08, 2015 at 04:51:36PM +0100, Kevin D wrote: > On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote: > > Hi, please CC me if that is not your usual fashion, because I am not > > subscribed. > > > > I use git for my build scripts - those are accessed over nfs. Since > > I started using 2.1 and later (I don't think I used 2.0) commands > > such as 'commit' take a long time before anything happens. I > > assumed that the newer version meant this would take longer. > > > > But today^Wyesterday I was bisecting the kernel on a local > > filesystem - even when the number of revisions left to test was in > > the single digits, git bisect took a long time to decide which > > revision should be the next one to test. The filesystems are ext4. > > Is this sort of delay normal now? > > > > What really prompted me to ask is that I ran git blame on a script, > > to see when I made a particular change so that I could add that > > information to a ticket, and almost gave up waiting because it felt > > as if it was taking for ever. > > > > What kind of repository are we talking about? How many files, how big? > Git should not have become significantly slower recently. > The comments on git bisect were for linus'skernel tree, on a local disk. 2.3GB of repo, just under 57000 files. My own repo of build scripts, where I have noticed the delay before git commit lets me type in the message, is an nfs v3 mount from another of my machines in the same room - ping between them gives times of 0.25 to 0.3 seconds and I think the nfs part is irrelevant. Here, the size is 70MB and 12133 files [ about 1500 scripts total, so the rest is from the commits ]. Some of this might be the drives - on the desktop with linus's tree the machine only supports SATA2 (3GB/S), but the machine serving my scripts goes back further and probably only supports SATA1 (1.5GB/S) > Also, might there be anti-virus software that slows down file access? No, this is all local access on linux machines. ĸen -- Nanny Ogg usually went to bed early. After all, she was an old lady. Sometimes she went to bed as early as 6 a.m. ``` ## Ken Moffat, 2015-03-08 19:20 Subject: Re: Git very slow ? Message-ID: <20150308192045.GB32504@milliways> URL: https://gitlist.dev/e/20150308192045.GB32504%40milliways In-Reply-To: <87zj7nmpdp.fsf@fencepost.gnu.org> ``` On Sun, Mar 08, 2015 at 05:21:22PM +0100, David Kastrup wrote: > Kevin D writes: > > > On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote: > >> Hi, please CC me if that is not your usual fashion, because I am not > >> subscribed. > >> > >> I use git for my build scripts - those are accessed over nfs. Since > >> I started using 2.1 and later (I don't think I used 2.0) commands > >> such as 'commit' take a long time before anything happens. I > >> assumed that the newer version meant this would take longer. > >> > >> But today^Wyesterday I was bisecting the kernel on a local > >> filesystem - even when the number of revisions left to test was in > >> the single digits, git bisect took a long time to decide which > >> revision should be the next one to test. The filesystems are ext4. > >> Is this sort of delay normal now? > >> > >> What really prompted me to ask is that I ran git blame on a script, > >> to see when I made a particular change so that I could add that > >> information to a ticket, and almost gave up waiting because it felt > >> as if it was taking for ever. > >> > > > > What kind of repository are we talking about? How many files, how big? > > Git should not have become significantly slower recently. > > Particularly not git-blame in 2.1. I should be quite surprised to see > any git-blame call running noticeably slower in 2.1 than in any > preceding version. > > What may have happened is that the repository recently got repacked > aggressively and thus any access to older revisions got slower. > However, that change would be mostly tied to the repository rather than > the version of Git you access it with. > That is possible - well, not recently-recently, but I might have repacked my repo of buildscripts some time last year. Running ls -al .git in that repository gives me: drwxr-xr-x 8 ken 100 4096 Mar 8 16:08 . drwxr-xr-x 48 ken 100 4096 Mar 8 03:05 .. -rw-r--r-- 1 ken 100 220 May 12 2014 BRANCH_DESCRIPTION drwxr-xr-x 2 ken 100 4096 Apr 13 2010 branches -rw-r--r-- 1 ken 100 470 Mar 8 16:08 COMMIT_EDITMSG -rw-r--r-- 1 ken 100 566 May 17 2014 config -rw-r--r-- 1 ken 100 73 May 1 2010 description -rw-r--r-- 1 ken 100 196439 Sep 17 21:56 gitk.cache -rw-rw-rw- 1 ken 100 29 Feb 8 22:19 HEAD drwxr-xr-x 2 ken 100 4096 May 1 2010 hooks -rw-r--r-- 1 ken 100 218255 Mar 8 16:07 index drwxr-xr-x 2 ken 100 4096 Sep 16 2013 info drwxr-xr-x 3 ken 100 4096 Sep 16 2013 logs drwxr-xr-x 260 ken 100 4096 Nov 12 2013 objects -rw-r--r-- 1 ken 100 41 Nov 11 06:05 ORIG_HEAD -rw-r--r-- 1 ken 100 1879 Sep 16 2013 packed-refs drwxr-xr-x 5 ken 100 4096 May 20 2014 refs -rw-r--r-- 1 ken 100 41 Dec 7 2010 RENAMED-REF Running git blame on a script which dates back to when the repo was created takes between 5 and 6 seconds to show the first screen, running it on a file first created last August took about 3 seconds. I have another small repo also on nfs with only a few files, started last year, and there git blame takes about a second. In my clone of linus' tree, running git blame scripts/headers.sh (a random example,I do not normally run git blame there) took 10 seconds to return the first screen. All of those timings are from preparing the comand, watching my desktop clock until it changes the second, and hitting enter - so, only accurate to approx one second. ĸen -- Nanny Ogg usually went to bed early. After all, she was an old lady. Sometimes she went to bed as early as 6 a.m. ``` ## David Kastrup, 2015-03-08 19:37 Subject: Re: Git very slow ? Message-ID: <87sidfmgag.fsf@fencepost.gnu.org> URL: https://gitlist.dev/e/87sidfmgag.fsf%40fencepost.gnu.org In-Reply-To: <20150308192045.GB32504@milliways> ``` Ken Moffat writes: > On Sun, Mar 08, 2015 at 05:21:22PM +0100, David Kastrup wrote: > >> Particularly not git-blame in 2.1. I should be quite surprised to see >> any git-blame call running noticeably slower in 2.1 than in any >> preceding version. >> >> What may have happened is that the repository recently got repacked >> aggressively and thus any access to older revisions got slower. >> However, that change would be mostly tied to the repository rather than >> the version of Git you access it with. >> > That is possible - well, not recently-recently, but I might have > repacked my repo of buildscripts some time last year. Running > ls -al .git > in that repository gives me: > drwxr-xr-x 8 ken 100 4096 Mar 8 16:08 . > drwxr-xr-x 48 ken 100 4096 Mar 8 03:05 .. > -rw-r--r-- 1 ken 100 220 May 12 2014 BRANCH_DESCRIPTION > drwxr-xr-x 2 ken 100 4096 Apr 13 2010 branches > -rw-r--r-- 1 ken 100 470 Mar 8 16:08 COMMIT_EDITMSG > -rw-r--r-- 1 ken 100 566 May 17 2014 config > -rw-r--r-- 1 ken 100 73 May 1 2010 description > -rw-r--r-- 1 ken 100 196439 Sep 17 21:56 gitk.cache > -rw-rw-rw- 1 ken 100 29 Feb 8 22:19 HEAD > drwxr-xr-x 2 ken 100 4096 May 1 2010 hooks > -rw-r--r-- 1 ken 100 218255 Mar 8 16:07 index > drwxr-xr-x 2 ken 100 4096 Sep 16 2013 info > drwxr-xr-x 3 ken 100 4096 Sep 16 2013 logs > drwxr-xr-x 260 ken 100 4096 Nov 12 2013 objects > -rw-r--r-- 1 ken 100 41 Nov 11 06:05 ORIG_HEAD > -rw-r--r-- 1 ken 100 1879 Sep 16 2013 packed-refs > drwxr-xr-x 5 ken 100 4096 May 20 2014 refs > -rw-r--r-- 1 ken 100 41 Dec 7 2010 RENAMED-REF > > Running git blame on a script which dates back to when the repo was > created takes between 5 and 6 seconds to show the first screen, Since git blame outputs everything once it is finished ("the first screen" is purely the pager's business), it needs to unpack the entire history of the file (unless no blameable lines remain at all) and look at it. 6 seconds tends not to be all that excessive for extracting more than 5 years of a file's history. -- David Kastrup ``` ## Linus Torvalds, 2015-03-08 19:39 Subject: Re: Git very slow ? Message-ID: URL: https://gitlist.dev/e/CA%2B55aFwn9vOWAL_9eHQt%2BkQN16j3%2BnQLeibWos86g5T_RZEizg%40mail.gmail.com In-Reply-To: <20150308190244.GA32504@milliways> ``` On Sun, Mar 8, 2015 at 12:02 PM, Ken Moffat wrote: > > The comments on git bisect were for linus'skernel tree, on a local > disk. 2.3GB of repo, just under 57000 files. Ugh. I hope you are talking about checked-out size. [torvalds@i7 linux]$ du -sh .git 850M .git because otherwise it sounds like that repo hasn't been repacked in forever. To really pack things (which can slow things down for old history as people said, but on the whole it tends to be a big win due to less IO), do git repack -adf --window=200 --depth=200 and go away for a while. Oh, and make sure your machine has enough memory and CPU to make that "for a while" not be *too* long. You should have a few hundred files (just a few tens of files directly after the repack) and that roughly 850MB of space for the repository information itself. But yeah, fully checked out and built with all the modules etc, you would have much more. That said, if you have something fairly that is consistently really slow (like the "git commit" you mentioned), *before* doing the repack, do strace -o ../trace-file -Ttt git commit and we can get a much better guess about why it's so slow. Send it to me in private email if you don't want to make it public, and I can take a look. > ping between them gives times of 0.25 to 0.3 seconds .. and I *really* hope that was not seconds, but ms. Otherwise your nfsv3 setup is going to be really really painful. Linus ``` ## Linus Torvalds, 2015-03-08 19:46 Subject: Re: Git very slow ? Message-ID: URL: https://gitlist.dev/e/CA%2B55aFzDRg4kHHGGHd91kVxfj8eX0g1w5T7SyN_CouCf%3D_tW3A%40mail.gmail.com In-Reply-To: <87sidfmgag.fsf@fencepost.gnu.org> ``` On Sun, Mar 8, 2015 at 12:37 PM, David Kastrup wrote: > > Since git blame outputs everything once it is finished ("the first > screen" is purely the pager's business), it needs to unpack the entire > history of the file (unless no blameable lines remain at all) and look > at it. 6 seconds tends not to be all that excessive for extracting more > than 5 years of a file's history. Yeah, "git blame" can easily be several seconds without anything being wrong. But "git commit" should be fairly instantaneous. Even over NFS. That said, on NFS in particular, make sure you don't have [core] PreloadIndex = false in your .gitconfig to disable the threaded index preloading. But "core.preloadindex" _should_ be enabled by default in anything but the most ancient git versions, and it can make a huge difference on NFS because it allows the 'lstat()' calls to check that the index is up-to-date to be done in parallel. Without that, git on NFS can be a bit sluggish. On local filesystems it normally doesn't make as much of a difference, since things tend to be either cached or seek-limited. Linus ``` ## Ken Moffat, 2015-03-08 23:31 Subject: Re: Git very slow ? Message-ID: <20150308233154.GB3792@milliways> URL: https://gitlist.dev/e/20150308233154.GB3792%40milliways In-Reply-To: ``` On Sun, Mar 08, 2015 at 12:39:10PM -0700, Linus Torvalds wrote: > On Sun, Mar 8, 2015 at 12:02 PM, Ken Moffat wrote: > > > > The comments on git bisect were for linus'skernel tree, on a local > > disk. 2.3GB of repo, just under 57000 files. > > Ugh. I hope you are talking about checked-out size. > > [torvalds@i7 linux]$ du -sh .git > 850M .git > > because otherwise it sounds like that repo hasn't been repacked in forever. > Yes - I had finished bisecting, with the build still present. > To really pack things (which can slow things down for old history as > people said, but on the whole it tends to be a big win due to less > IO), do > > git repack -adf --window=200 --depth=200 > > and go away for a while. Oh, and make sure your machine has enough > memory and CPU to make that "for a while" not be *too* long. > For that, many thanks - this desktop has about 7GB (integrated graphics steal a bit), current AMD desktop, and the repack of my scripts repo took about 56 seconds. I'll do that on my copy of the kernel tomorrow (it's on another machine). > You should have a few hundred files (just a few tens of files directly > after the repack) and that roughly 850MB of space for the repository > information itself. > > But yeah, fully checked out and built with all the modules etc, you > would have much more. > > That said, if you have something fairly that is consistently really > slow (like the "git commit" you mentioned), *before* doing the repack, > do > > strace -o ../trace-file -Ttt git commit > > and we can get a much better guess about why it's so slow. Send it to > me in private email if you don't want to make it public, and I can > take a look. I don't think you need to look - it was taking most of the time (about 8 seconds) looping through the many files below .git/objects. The trace was just over 9000 lines, repeating after the repack was less than 1300 lines. It's available (97K after using xz) if you think it would be useful, but I think you have already diagnosed the problem and solution. > > > ping between them gives times of 0.25 to 0.3 seconds > > .. and I *really* hope that was not seconds, but ms. Otherwise your > nfsv3 setup is going to be really really painful. > > Linus Yes, I'm not always good at knowing the right units. Thanks for the help. ĸen -- Nanny Ogg usually went to bed early. After all, she was an old lady. Sometimes she went to bed as early as 6 a.m. ```