# git blame performance 3 messages from 2015-11-06 to 2015-11-06. Participants: Jan Smets, Michael Haggerty, Junio C Hamano. Thread: https://gitlist.dev/t/40739 ## Jan Smets, 2015-11-06 13:37 Subject: git blame performance Message-ID: <563CAD30.6040608@alcatel-lucent.com> URL: https://gitlist.dev/e/563CAD30.6040608%40alcatel-lucent.com ``` Hi I have recently migrated a fairly large project from CVS to Git. One of the issues we're having is the blame/annotate performance. The repository contains +650k commits total, of which ~300k are on master. (raw size = ~8GB) Running blame on one of the oldest files takes over 30 seconds. This is on a fairly beefy (server) machine with lots of ram and the repository on a ramdisk. Running git 2.5.2 cvs annotate of the same file (over the network) is ready in 0.8 seconds. blame/annotate is a frequently used operation, ranging between 5 to 20 usages a day per developer. I have two questions 1) Is there a way to speed this up (in git)? eg: can it run multi threaded? build pre-cached blame views? 2) I was thinking to work around the issue and use gitweb blame_incremental and pre-populate the cache. If you can think of any other (short term) solutions I would really like to hear them. Thank you - Jan ``` ## Michael Haggerty, 2015-11-06 14:52 Subject: Re: git blame performance Message-ID: <563CBEC8.7070209@alum.mit.edu> URL: https://gitlist.dev/e/563CBEC8.7070209%40alum.mit.edu In-Reply-To: <563CAD30.6040608@alcatel-lucent.com> ``` On 11/06/2015 02:37 PM, Jan Smets wrote: > I have recently migrated a fairly large project from CVS to Git. > One of the issues we're having is the blame/annotate performance. > [...] > cvs annotate of the same file (over the network) is ready in 0.8 seconds. > blame/annotate is a frequently used operation, ranging between 5 to 20 > usages a day per developer. cvs annotate and git blame both have to follow history back until they find the commit that introduced the oldest line that is still in the current version of the file. So for a really old file, a lot of history has to be walked through. The reason that cvs annotate is so much faster than git blame is that CVS stores revisions filewise, with all of the modifications to file $FILE being stored in a single $FILE,v file. So in the worst case, CVS only has to read this one file. Git, on the other hand, stores revisions treewise. It has no way of knowing, ab initio, which revisions touched a given file. (In fact, this concept is not even well-defined because the answer depends on things like whether copy (-C) and move (-M) detection are turned on and what parameters they were given.) This means that git blame has to traverse most of history to find the commits that touched $FILE. Slow git blame is thus a relatively unavoidable consequence of Git's data model. That's not to say that it can't be sped up somewhat, but it will never reach CVS speeds. But it does have some features that can reduce the work: -L ,, -L : -- Annotate only the given line range. This option can speed things up (1) if the range of lines does not include the oldest lines, (2) by limiting which parents of merge commits have to be followed. --incremental -- if you are using this command to build tooling, this option allows partial results to be returned early, to reduce the wait until the user sees something. If you are not interested in changes older than a certain date or revision, you can limit the amount of history that git blame traverses. See SPECIFYING RANGES in the manpage. Michael -- Michael Haggerty mhagger@alum.mit.edu ``` ## Junio C Hamano, 2015-11-06 17:53 Subject: Re: git blame performance Message-ID: URL: https://gitlist.dev/e/xmqqwptv2eve.fsf%40gitster.mtv.corp.google.com In-Reply-To: <563CBEC8.7070209@alum.mit.edu> ``` Michael Haggerty writes: > The reason that cvs annotate is so much faster than git blame is that > CVS stores revisions filewise, with all of the modifications to file > $FILE being stored in a single $FILE,v file. So in the worst case, CVS > only has to read this one file. > > Git, on the other hand, stores revisions treewise. It has no way of > knowing, ab initio, which revisions touched a given file. (In fact, this > concept is not even well-defined because the answer depends on things > like whether copy (-C) and move (-M) detection are turned on and what > parameters they were given.) This means that git blame has to traverse > most of history to find the commits that touched $FILE. > > Slow git blame is thus a relatively unavoidable consequence of Git's > data model. That's not to say that it can't be sped up somewhat, but it > will never reach CVS speeds. Another thing to consider for a converted repository is that mass converters tend to either not make a pack at all or make a pack that is horribly inefficient to access. Running "git repack -a -d -f" with a small value of "--depth" may be a thing worth trying, if that is contributing to the performance. ```