How to efficiently blame an entire repo?

Let's say you've got a repo with ~ 40K files and 35K commits. Well-packed .git is about 800MB.

You want to find out how many lines of code a particular group of individuals has contributed to HEAD.

The naive solution is to run git blame on all 40K files grep'ing for the just the authors you want.

Possibly a step up from that is first using log --name-status --author=... to find just the files which have been touched by those authors and then blaming only those files.

I guess the next step up would be parsing the diff hunks output by log -p, but then you're basically re-implementing blame I think.

Am I missing a clever solution?

j.

On Thu, Apr 29, 2010 at 7:12 PM, Jay Soffian <jaysoffian@gmail.com> wrote:

How often do you need to do this? If it's just once in your life, then the brute force solution of just letting 'git blame' grind through it for a few hours is probably the cleverest :)

Have fun,

Avery

On Fri, Apr 30, 2010 at 3:45 PM, Avery Pennarun <apenwarr@gmail.com> wrote:

Yeah, I ended up doing this basically.

Setup a .mailmap mapping the authors I was interested in to domain.com. Then:

$ git log --pretty='%H %aE' HEAD | grep domain.com |
  awk '{print $1}' |
  git log --no-walk --stdin --name-only --pretty=%n |
  grep -v '^$' | sort -u > files1
$ git ls-files | sort > files2
$ comm -12 files1 files2 > files
$ xargs < files -n1 git annotate | grep domain.com

I didn't use --author=domain.com w/the first log invocation because I wasn't sure if it respected .mailmap and was too lazy to look it up.

I probably I could've used --diff-filter in the second log invocation, but, meh.

So that worked. Took about 12 minutes to run on a recent Macbook Pro. Aside, blame's --porcelain switch is rather poorly documented and annotate seemed to have the right output for the job.

j.

On Thu, Apr 29, 2010 at 07:12:27PM -0400, Jay Soffian wrote:

With the exception of your "blame only those files that you know your authors have touched" optimization, I think you pretty much have to do this. Anything else will just be reimplementing blame. You can't throw away most content prematurely, because it may end up blaming to your authors of interest eventually.

I think this is also what Junio ended up doing when presenting at GitTogether '08:

  http://userweb.kernel.org/~junio/200810-Chron.pdf

In theory you might be able to do multi-file blame faster. I would be curious to see the performance difference between:

  $ git blame file1 file2 ;# not actually implemented

and

  $ for i in file1 file2; do git blame $i; done

Much of the work is O(content), but there is some overlap in walking the history and generating diffs.

-Peff