threads / discuss / 38736

Git very slow ?

Subject: Git very slow ?

## tl;dr

9 messages between Mar 7, 2015 and Mar 8, 2015.

replies: 8people: 4as markdown or json

Ken Moffat· Mar 7, 2015, 01:30 UTC · lore

Hi, please CC me if that is not your usual fashion, because I am not subscribed.

I use git for my build scripts - those are accessed over nfs. Since I started using 2.1 and later (I don't think I used 2.0) commands such as 'commit' take a long time before anything happens. I assumed that the newer version meant this would take longer.

But today^Wyesterday I was bisecting the kernel on a local filesystem - even when the number of revisions left to test was in the single digits, git bisect took a long time to decide which revision should be the next one to test. The filesystems are ext4. Is this sort of delay normal now?

What really prompted me to ask is that I ran git blame on a script, to see when I made a particular change so that I could add that information to a ticket, and almost gave up waiting because it felt as if it was taking for ever.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.
Kevin D· Mar 8, 2015, 15:51 UTC · re: Ken Moffat · lore

Re: Git very slow ?

On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:
Show 19 quoted lines
> Hi, please CC me if that is not your usual fashion, because I am not
> subscribed.
> 
> I use git for my build scripts - those are accessed over nfs.  Since
> I started using 2.1 and later (I don't think I used 2.0) commands
> such as 'commit' take a long time before anything happens.  I
> assumed that the newer version meant this would take longer.
> 
> But today^Wyesterday I was bisecting the kernel on a local
> filesystem - even when the number of revisions left to test was in
> the single digits, git bisect took a long time to decide which
> revision should be the next one to test.  The filesystems are ext4.
> Is this sort of delay normal now?
> 
> What really prompted me to ask is that I ran git blame on a script,
> to see when I made a particular change so that I could add that
> information to a ticket, and almost gave up waiting because it felt
> as if it was taking for ever.
>

What kind of repository are we talking about? How many files, how big? Git should not have become significantly slower recently.

Also, might there be anti-virus software that slows down file access?
David Kastrup· Mar 8, 2015, 16:21 UTC · re: Kevin D · lore

Re: Git very slow ?

Kevin D <me@ikke.info> writes:
Show 23 quoted lines
> On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:
>> Hi, please CC me if that is not your usual fashion, because I am not
>> subscribed.
>> 
>> I use git for my build scripts - those are accessed over nfs.  Since
>> I started using 2.1 and later (I don't think I used 2.0) commands
>> such as 'commit' take a long time before anything happens.  I
>> assumed that the newer version meant this would take longer.
>> 
>> But today^Wyesterday I was bisecting the kernel on a local
>> filesystem - even when the number of revisions left to test was in
>> the single digits, git bisect took a long time to decide which
>> revision should be the next one to test.  The filesystems are ext4.
>> Is this sort of delay normal now?
>> 
>> What really prompted me to ask is that I ran git blame on a script,
>> to see when I made a particular change so that I could add that
>> information to a ticket, and almost gave up waiting because it felt
>> as if it was taking for ever.
>> 
>
> What kind of repository are we talking about? How many files, how big?
> Git should not have become significantly slower recently.

Particularly not git-blame in 2.1. I should be quite surprised to see any git-blame call running noticeably slower in 2.1 than in any preceding version.

What may have happened is that the repository recently got repacked aggressively and thus any access to older revisions got slower. However, that change would be mostly tied to the repository rather than the version of Git you access it with.

-- 
David Kastrup
Ken Moffat· Mar 8, 2015, 19:20 UTC · re: David Kastrup · lore

Re: Git very slow ?

On Sun, Mar 08, 2015 at 05:21:22PM +0100, David Kastrup wrote:
Show 35 quoted lines
> Kevin D <me@ikke.info> writes:
> 
> > On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:
> >> Hi, please CC me if that is not your usual fashion, because I am not
> >> subscribed.
> >> 
> >> I use git for my build scripts - those are accessed over nfs.  Since
> >> I started using 2.1 and later (I don't think I used 2.0) commands
> >> such as 'commit' take a long time before anything happens.  I
> >> assumed that the newer version meant this would take longer.
> >> 
> >> But today^Wyesterday I was bisecting the kernel on a local
> >> filesystem - even when the number of revisions left to test was in
> >> the single digits, git bisect took a long time to decide which
> >> revision should be the next one to test.  The filesystems are ext4.
> >> Is this sort of delay normal now?
> >> 
> >> What really prompted me to ask is that I ran git blame on a script,
> >> to see when I made a particular change so that I could add that
> >> information to a ticket, and almost gave up waiting because it felt
> >> as if it was taking for ever.
> >> 
> >
> > What kind of repository are we talking about? How many files, how big?
> > Git should not have become significantly slower recently.
> 
> Particularly not git-blame in 2.1.  I should be quite surprised to see
> any git-blame call running noticeably slower in 2.1 than in any
> preceding version.
> 
> What may have happened is that the repository recently got repacked
> aggressively and thus any access to older revisions got slower.
> However, that change would be mostly tied to the repository rather than
> the version of Git you access it with.
> 
That is possible - well, not recently-recently, but I might have
repacked my repo of buildscripts some time last year.  Running
 ls -al .git
in that repository gives me:
drwxr-xr-x   8 ken 100   4096 Mar  8 16:08 .
drwxr-xr-x  48 ken 100   4096 Mar  8 03:05 ..
-rw-r--r--   1 ken 100    220 May 12  2014 BRANCH_DESCRIPTION
drwxr-xr-x   2 ken 100   4096 Apr 13  2010 branches
-rw-r--r--   1 ken 100    470 Mar  8 16:08 COMMIT_EDITMSG
-rw-r--r--   1 ken 100    566 May 17  2014 config
-rw-r--r--   1 ken 100     73 May  1  2010 description
-rw-r--r--   1 ken 100 196439 Sep 17 21:56 gitk.cache
-rw-rw-rw-   1 ken 100     29 Feb  8 22:19 HEAD
drwxr-xr-x   2 ken 100   4096 May  1  2010 hooks
-rw-r--r--   1 ken 100 218255 Mar  8 16:07 index
drwxr-xr-x   2 ken 100   4096 Sep 16  2013 info
drwxr-xr-x   3 ken 100   4096 Sep 16  2013 logs
drwxr-xr-x 260 ken 100   4096 Nov 12  2013 objects
-rw-r--r--   1 ken 100     41 Nov 11 06:05 ORIG_HEAD
-rw-r--r--   1 ken 100   1879 Sep 16  2013 packed-refs
drwxr-xr-x   5 ken 100   4096 May 20  2014 refs
-rw-r--r--   1 ken 100     41 Dec  7  2010 RENAMED-REF

Running git blame on a script which dates back to when the repo was created takes between 5 and 6 seconds to show the first screen, running it on a file first created last August took about 3 seconds. I have another small repo also on nfs with only a few files, started last year, and there git blame takes about a second.

In my clone of linus' tree, running git blame scripts/headers.sh (a random example,I do not normally run git blame there) took 10 seconds to return the first screen.

All of those timings are from preparing the comand, watching my desktop clock until it changes the second, and hitting enter - so, only accurate to approx one second.

ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.
David Kastrup· Mar 8, 2015, 19:37 UTC · re: Ken Moffat · lore

Re: Git very slow ?

Ken Moffat <zarniwhoop@ntlworld.com> writes:
Show 36 quoted lines
> On Sun, Mar 08, 2015 at 05:21:22PM +0100, David Kastrup wrote:
>
>> Particularly not git-blame in 2.1.  I should be quite surprised to see
>> any git-blame call running noticeably slower in 2.1 than in any
>> preceding version.
>> 
>> What may have happened is that the repository recently got repacked
>> aggressively and thus any access to older revisions got slower.
>> However, that change would be mostly tied to the repository rather than
>> the version of Git you access it with.
>> 
> That is possible - well, not recently-recently, but I might have
> repacked my repo of buildscripts some time last year.  Running
>  ls -al .git
> in that repository gives me:
> drwxr-xr-x   8 ken 100   4096 Mar  8 16:08 .
> drwxr-xr-x  48 ken 100   4096 Mar  8 03:05 ..
> -rw-r--r--   1 ken 100    220 May 12  2014 BRANCH_DESCRIPTION
> drwxr-xr-x   2 ken 100   4096 Apr 13  2010 branches
> -rw-r--r--   1 ken 100    470 Mar  8 16:08 COMMIT_EDITMSG
> -rw-r--r--   1 ken 100    566 May 17  2014 config
> -rw-r--r--   1 ken 100     73 May  1  2010 description
> -rw-r--r--   1 ken 100 196439 Sep 17 21:56 gitk.cache
> -rw-rw-rw-   1 ken 100     29 Feb  8 22:19 HEAD
> drwxr-xr-x   2 ken 100   4096 May  1  2010 hooks
> -rw-r--r--   1 ken 100 218255 Mar  8 16:07 index
> drwxr-xr-x   2 ken 100   4096 Sep 16  2013 info
> drwxr-xr-x   3 ken 100   4096 Sep 16  2013 logs
> drwxr-xr-x 260 ken 100   4096 Nov 12  2013 objects
> -rw-r--r--   1 ken 100     41 Nov 11 06:05 ORIG_HEAD
> -rw-r--r--   1 ken 100   1879 Sep 16  2013 packed-refs
> drwxr-xr-x   5 ken 100   4096 May 20  2014 refs
> -rw-r--r--   1 ken 100     41 Dec  7  2010 RENAMED-REF
>
> Running git blame on a script which dates back to when the repo was
> created takes between 5 and 6 seconds to show the first screen,

Since git blame outputs everything once it is finished ("the first screen" is purely the pager's business), it needs to unpack the entire history of the file (unless no blameable lines remain at all) and look at it. 6 seconds tends not to be all that excessive for extracting more than 5 years of a file's history.

-- 
David Kastrup
Linus Torvalds· Mar 8, 2015, 19:46 UTC · re: David Kastrup · lore

Re: Git very slow ?

On Sun, Mar 8, 2015 at 12:37 PM, David Kastrup <dak@gnu.org> wrote:
Show 6 quoted lines
>
> Since git blame outputs everything once it is finished ("the first
> screen" is purely the pager's business), it needs to unpack the entire
> history of the file (unless no blameable lines remain at all) and look
> at it.  6 seconds tends not to be all that excessive for extracting more
> than 5 years of a file's history.
Yeah, "git blame" can easily be several seconds without anything being wrong.
But "git commit" should be fairly instantaneous. Even over NFS.
That said, on NFS in particular, make sure you don't have
    [core]
        PreloadIndex = false
in your .gitconfig to disable the threaded index preloading.

But "core.preloadindex" _should_ be enabled by default in anything but the most ancient git versions, and it can make a huge difference on NFS because it allows the 'lstat()' calls to check that the index is up-to-date to be done in parallel. Without that, git on NFS can be a bit sluggish.

On local filesystems it normally doesn't make as much of a difference, since things tend to be either cached or seek-limited.

                             Linus
Ken Moffat· Mar 8, 2015, 19:02 UTC · re: Kevin D · lore

Re: Git very slow ?

On Sun, Mar 08, 2015 at 04:51:36PM +0100, Kevin D wrote:
Show 24 quoted lines
> On Sat, Mar 07, 2015 at 01:30:07AM +0000, Ken Moffat wrote:
> > Hi, please CC me if that is not your usual fashion, because I am not
> > subscribed.
> > 
> > I use git for my build scripts - those are accessed over nfs.  Since
> > I started using 2.1 and later (I don't think I used 2.0) commands
> > such as 'commit' take a long time before anything happens.  I
> > assumed that the newer version meant this would take longer.
> > 
> > But today^Wyesterday I was bisecting the kernel on a local
> > filesystem - even when the number of revisions left to test was in
> > the single digits, git bisect took a long time to decide which
> > revision should be the next one to test.  The filesystems are ext4.
> > Is this sort of delay normal now?
> > 
> > What really prompted me to ask is that I ran git blame on a script,
> > to see when I made a particular change so that I could add that
> > information to a ticket, and almost gave up waiting because it felt
> > as if it was taking for ever.
> > 
> 
> What kind of repository are we talking about? How many files, how big?
> Git should not have become significantly slower recently.
>

The comments on git bisect were for linus'skernel tree, on a local disk. 2.3GB of repo, just under 57000 files.

My own repo of build scripts, where I have noticed the delay before git commit lets me type in the message, is an nfs v3 mount from another of my machines in the same room - ping between them gives times of 0.25 to 0.3 seconds and I think the nfs part is irrelevant. Here, the size is 70MB and 12133 files [ about 1500 scripts total, so the rest is from the commits ].

Some of this might be the drives - on the desktop with linus's tree the machine only supports SATA2 (3GB/S), but the machine serving my scripts goes back further and probably only supports SATA1 (1.5GB/S)

> Also, might there be anti-virus software that slows down file access?
No, this is all local access on linux machines.
ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.
Linus Torvalds· Mar 8, 2015, 19:39 UTC · re: Ken Moffat · lore

Re: Git very slow ?

On Sun, Mar 8, 2015 at 12:02 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
>
> The comments on git bisect were for linus'skernel tree, on a local
> disk.  2.3GB of repo, just under 57000 files.
Ugh. I hope you are talking about checked-out size.
    [torvalds@i7 linux]$ du -sh .git
   850M .git
because otherwise it sounds like that repo hasn't been repacked in forever.

To really pack things (which can slow things down for old history as people said, but on the whole it tends to be a big win due to less IO), do

   git repack -adf --window=200 --depth=200

and go away for a while. Oh, and make sure your machine has enough memory and CPU to make that "for a while" not be *too* long.

You should have a few hundred files (just a few tens of files directly after the repack) and that roughly 850MB of space for the repository information itself.

But yeah, fully checked out and built with all the modules etc, you would have much more.

That said, if you have something fairly that is consistently really slow (like the "git commit" you mentioned), *before* doing the repack, do

   strace -o ../trace-file -Ttt git commit

and we can get a much better guess about why it's so slow. Send it to me in private email if you don't want to make it public, and I can take a look.

> ping between them gives times of 0.25 to 0.3 seconds

.. and I *really* hope that was not seconds, but ms. Otherwise your nfsv3 setup is going to be really really painful.

                          Linus
Ken Moffat· Mar 8, 2015, 23:31 UTC · re: Linus Torvalds · lore

Re: Git very slow ?

On Sun, Mar 08, 2015 at 12:39:10PM -0700, Linus Torvalds wrote:
Show 12 quoted lines
> On Sun, Mar 8, 2015 at 12:02 PM, Ken Moffat <zarniwhoop@ntlworld.com> wrote:
> >
> > The comments on git bisect were for linus'skernel tree, on a local
> > disk.  2.3GB of repo, just under 57000 files.
> 
> Ugh. I hope you are talking about checked-out size.
> 
>     [torvalds@i7 linux]$ du -sh .git
>    850M .git
> 
> because otherwise it sounds like that repo hasn't been repacked in forever.
> 
 Yes - I had finished bisecting, with the build still present.
Show 9 quoted lines
> To really pack things (which can slow things down for old history as
> people said, but on the whole it tends to be a big win due to less
> IO), do
> 
>    git repack -adf --window=200 --depth=200
> 
> and go away for a while. Oh, and make sure your machine has enough
> memory and CPU to make that "for a while" not be *too* long.
> 
 For that, many thanks - this desktop has about 7GB (integrated
graphics steal a bit), current AMD desktop, and the repack of my
scripts repo took about 56 seconds.  I'll do that on my copy of
the kernel tomorrow (it's on another machine).
Show 16 quoted lines
> You should have a few hundred files (just a few tens of files directly
> after the repack) and that roughly 850MB of space for the repository
> information itself.
> 
> But yeah, fully checked out and built with all the modules etc, you
> would have much more.
> 
> That said, if you have something fairly that is consistently really
> slow (like the "git commit" you mentioned), *before* doing the repack,
> do
> 
>    strace -o ../trace-file -Ttt git commit
> 
> and we can get a much better guess about why it's so slow. Send it to
> me in private email if you don't want to make it public, and I can
> take a look.
 I don't think you need to look - it was taking most of the time
(about 8 seconds) looping through the many files below .git/objects.
The trace was just over 9000 lines, repeating after the repack was
less than 1300 lines.  It's available (97K after using xz) if you
think it would be useful, but I think you have already diagnosed the
problem and solution.
Show 7 quoted lines
> 
> > ping between them gives times of 0.25 to 0.3 seconds
> 
> .. and I *really* hope that was not seconds, but ms. Otherwise your
> nfsv3 setup is going to be really really painful.
> 
>                           Linus
 Yes, I'm not always good at knowing the right units.  Thanks for the
help.
ĸen
-- 
Nanny Ogg usually went to bed early. After all, she was an old lady.
Sometimes she went to bed as early as 6 a.m.

← back to recent threads