{"thread":{"id":"40739","subject":"git blame performance","startedAt":"2015-11-06T13:37:52Z","lastAt":"2015-11-06T17:53:57Z","messageCount":3,"participants":["Jan Smets","Michael Haggerty","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"273015","messageId":"563CAD30.6040608@alcatel-lucent.com","threadId":"40739","inReplyTo":null,"subject":"git blame performance","fromName":"Jan Smets","fromEmail":"jan.smets@alcatel-lucent.com","sentAt":"2015-11-06T13:37:52Z","receivedAt":"2015-11-06T13:37:52Z","isPatch":false,"sender":{"key":"jan.smets@alcatel-lucent.com","avatar":null},"body":"Hi\n\nI have recently migrated a fairly large project from CVS to Git.\nOne of the issues we're having is the blame/annotate performance.\n\nThe repository contains +650k commits total, of which ~300k are on \nmaster. (raw size = ~8GB)\n\nRunning blame on one of the oldest files takes over 30 seconds.\nThis is on a fairly beefy (server) machine with lots of ram and the \nrepository on a ramdisk. Running git 2.5.2\n\ncvs annotate of the same file (over the network) is ready in 0.8 seconds.\nblame/annotate is a frequently used operation, ranging between 5 to 20 \nusages a day per developer.\n\nI have two questions\n\n 1) Is there a way to speed this up (in git)? eg: can it run multi \nthreaded? build pre-cached blame views?\n 2) I was thinking to work around the issue and use gitweb \nblame_incremental and pre-populate the cache.\n\nIf you can think of any other (short term) solutions I would really like \nto hear them.\n\nThank you\n\n- Jan\n"},{"id":"273021","messageId":"563CBEC8.7070209@alum.mit.edu","threadId":"40739","inReplyTo":"563CAD30.6040608@alcatel-lucent.com","subject":"Re: git blame performance","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2015-11-06T14:52:56Z","receivedAt":"2015-11-06T14:52:56Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 11/06/2015 02:37 PM, Jan Smets wrote:\n> I have recently migrated a fairly large project from CVS to Git.\n> One of the issues we're having is the blame/annotate performance.\n> [...]\n> cvs annotate of the same file (over the network) is ready in 0.8 seconds.\n> blame/annotate is a frequently used operation, ranging between 5 to 20\n> usages a day per developer.\n\ncvs annotate and git blame both have to follow history back until they\nfind the commit that introduced the oldest line that is still in the\ncurrent version of the file. So for a really old file, a lot of history\nhas to be walked through.\n\nThe reason that cvs annotate is so much faster than git blame is that\nCVS stores revisions filewise, with all of the modifications to file\n$FILE being stored in a single $FILE,v file. So in the worst case, CVS\nonly has to read this one file.\n\nGit, on the other hand, stores revisions treewise. It has no way of\nknowing, ab initio, which revisions touched a given file. (In fact, this\nconcept is not even well-defined because the answer depends on things\nlike whether copy (-C) and move (-M) detection are turned on and what\nparameters they were given.) This means that git blame has to traverse\nmost of history to find the commits that touched $FILE.\n\nSlow git blame is thus a relatively unavoidable consequence of Git's\ndata model. That's not to say that it can't be sped up somewhat, but it\nwill never reach CVS speeds.\n\nBut it does have some features that can reduce the work:\n\n-L ,, -L : -- Annotate only the given line range.\nThis option can speed things up (1) if the range of lines does not\ninclude the oldest lines, (2) by limiting which parents of merge commits\nhave to be followed.\n\n--incremental -- if you are using this command to build tooling, this\noption allows partial results to be returned early, to reduce the wait\nuntil the user sees something.\n\nIf you are not interested in changes older than a certain date or\nrevision, you can limit the amount of history that git blame traverses.\nSee SPECIFYING RANGES in the manpage.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\n"},{"id":"273025","messageId":"xmqqwptv2eve.fsf@gitster.mtv.corp.google.com","threadId":"40739","inReplyTo":"563CBEC8.7070209@alum.mit.edu","subject":"Re: git blame performance","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-11-06T17:53:57Z","receivedAt":"2015-11-06T17:53:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michael Haggerty writes:\n\n> The reason that cvs annotate is so much faster than git blame is that\n> CVS stores revisions filewise, with all of the modifications to file\n> $FILE being stored in a single $FILE,v file. So in the worst case, CVS\n> only has to read this one file.\n>\n> Git, on the other hand, stores revisions treewise. It has no way of\n> knowing, ab initio, which revisions touched a given file. (In fact, this\n> concept is not even well-defined because the answer depends on things\n> like whether copy (-C) and move (-M) detection are turned on and what\n> parameters they were given.) This means that git blame has to traverse\n> most of history to find the commits that touched $FILE.\n>\n> Slow git blame is thus a relatively unavoidable consequence of Git's\n> data model. That's not to say that it can't be sped up somewhat, but it\n> will never reach CVS speeds.\n\nAnother thing to consider for a converted repository is that mass\nconverters tend to either not make a pack at all or make a pack that\nis horribly inefficient to access. Running \"git repack -a -d -f\"\nwith a small value of \"--depth\" may be a thing worth trying, if that\nis contributing to the performance.\n"}]}