{"thread":{"id":"23636","subject":"How to efficiently blame an entire repo?","startedAt":"2010-04-29T23:12:27Z","lastAt":"2010-04-30T21:21:03Z","messageCount":4,"participants":["Jay Soffian","Avery Pennarun","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"140638","messageId":"r2l76718491004291612qbe0834ela89f0606e5f477e2@mail.gmail.com","threadId":"23636","inReplyTo":null,"subject":"How to efficiently blame an entire repo?","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2010-04-29T23:12:27Z","receivedAt":"2010-04-29T23:12:27Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"Let's say you've got a repo with ~ 40K files and 35K commits.\nWell-packed .git is about 800MB.\n\nYou want to find out how many lines of code a particular group of\nindividuals has contributed to HEAD.\n\nThe naive solution is to run git blame on all 40K files grep'ing for\nthe just the authors you want.\n\nPossibly a step up from that is first using log --name-status\n--author=... to find just the files which have been touched by those\nauthors and then blaming only those files.\n\nI guess the next step up would be parsing the diff hunks output by log\n-p, but then you're basically re-implementing blame I think.\n\nAm I missing a clever solution?\n\nj.\n"},{"id":"140653","messageId":"x2s32541b131004301245o46d8e855va0b51ebce279398b@mail.gmail.com","threadId":"23636","inReplyTo":"r2l76718491004291612qbe0834ela89f0606e5f477e2@mail.gmail.com","subject":"Re: How to efficiently blame an entire repo?","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-04-30T19:45:12Z","receivedAt":"2010-04-30T19:45:12Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Thu, Apr 29, 2010 at 7:12 PM, Jay Soffian wrote:\n> Let's say you've got a repo with ~ 40K files and 35K commits.\n> Well-packed .git is about 800MB.\n>\n> You want to find out how many lines of code a particular group of\n> individuals has contributed to HEAD.\n>[...]\n> Am I missing a clever solution?\n\nHow often do you need to do this? If it's just once in your life,\nthen the brute force solution of just letting 'git blame' grind\nthrough it for a few hours is probably the cleverest :)\n\nHave fun,\n\nAvery\n"},{"id":"140656","messageId":"o2m76718491004301316tfbaf434alb94e6bfd001d36b6@mail.gmail.com","threadId":"23636","inReplyTo":"x2s32541b131004301245o46d8e855va0b51ebce279398b@mail.gmail.com","subject":"Re: How to efficiently blame an entire repo?","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2010-04-30T20:16:13Z","receivedAt":"2010-04-30T20:16:13Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"On Fri, Apr 30, 2010 at 3:45 PM, Avery Pennarun wrote:\n> On Thu, Apr 29, 2010 at 7:12 PM, Jay Soffian wrote:\n>> Let's say you've got a repo with ~ 40K files and 35K commits.\n>> Well-packed .git is about 800MB.\n>>\n>> You want to find out how many lines of code a particular group of\n>> individuals has contributed to HEAD.\n>>[...]\n>> Am I missing a clever solution?\n>\n> How often do you need to do this?  If it's just once in your life,\n> then the brute force solution of just letting 'git blame' grind\n> through it for a few hours is probably the cleverest :)\n\nYeah, I ended up doing this basically.\n\nSetup a .mailmap mapping the authors I was interested in to domain.com. Then:\n\n$ git log --pretty='%H %aE' HEAD | grep domain.com |\n awk '{print $1}' |\n git log --no-walk --stdin --name-only --pretty=%n |\n grep -v '^$' | sort -u > files1\n$ git ls-files | sort > files2\n$ comm -12 files1 files2 > files\n$ xargs < files -n1 git annotate | grep domain.com\n\nI didn't use --author=domain.com w/the first log invocation because I\nwasn't sure if it respected .mailmap and was too lazy to look it up.\n\nI probably I could've used --diff-filter in the second log invocation, but, meh.\n\nSo that worked. Took about 12 minutes to run on a recent Macbook Pro.\nAside, blame's --porcelain switch is rather poorly documented and\nannotate seemed to have the right output for the job.\n\nj.\n"},{"id":"140658","messageId":"20100430212103.GB14039@coredump.intra.peff.net","threadId":"23636","inReplyTo":"r2l76718491004291612qbe0834ela89f0606e5f477e2@mail.gmail.com","subject":"Re: How to efficiently blame an entire repo?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2010-04-30T21:21:03Z","receivedAt":"2010-04-30T21:21:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 29, 2010 at 07:12:27PM -0400, Jay Soffian wrote:\n\n> Let's say you've got a repo with ~ 40K files and 35K commits.\n> Well-packed .git is about 800MB.\n> \n> You want to find out how many lines of code a particular group of\n> individuals has contributed to HEAD.\n> \n> The naive solution is to run git blame on all 40K files grep'ing for\n> the just the authors you want.\n\nWith the exception of your \"blame only those files that you know your\nauthors have touched\" optimization, I think you pretty much have to do\nthis. Anything else will just be reimplementing blame. You can't throw\naway most content prematurely, because it may end up blaming to your\nauthors of interest eventually.\n\nI think this is also what Junio ended up doing when presenting at\nGitTogether '08:\n\n http://userweb.kernel.org/~junio/200810-Chron.pdf\n\nIn theory you might be able to do multi-file blame faster. I would be\ncurious to see the performance difference between:\n\n $ git blame file1 file2 ;# not actually implemented\n\nand\n\n $ for i in file1 file2; do git blame $i; done\n\nMuch of the work is O(content), but there is some overlap in walking the\nhistory and generating diffs.\n\n-Peff\n"}]}