{"thread":{"id":"41679","subject":"Graph sloc tool for git repos","startedAt":"2016-03-12T11:00:26Z","lastAt":"2016-03-14T02:50:53Z","messageCount":3,"participants":["Kai Hendry","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"280679","messageId":"1457780426.2632189.547083938.25305E83@webmail.messagingengine.com","threadId":"41679","inReplyTo":null,"subject":"Graph sloc tool for git repos","fromName":"Kai Hendry","fromEmail":"hendry@iki.fi","sentAt":"2016-03-12T11:00:26Z","receivedAt":"2016-03-12T11:00:26Z","isPatch":false,"sender":{"key":"hendry@iki.fi","avatar":null},"body":"Hi there,\n\nI penned a script to plot SLOC of a git project using GNUplot & I\nthought the fastest way to count code fluctuations was via `git show\n--numstat`.\n\nHowever that requires some awk counting of the lines:\nhttps://github.com/kaihendry/graphsloc/blob/5f31e388e9b655e1801f13885f4311d221663a19/collect-stats.sh#L32\n\nIs there a better way I missed? I think there is bug since my graph was\na factor of 10 out whilst graphing Linux:\nhttps://twitter.com/kaihendry/status/706627679924174848\n\n(Though the shape looks right)\n\nGood news it's generating graphs for smaller projects just fine:\nhttp://s.natalian.org/2016-03-12/dwm-3465bed.csv.svg\n\nAnyway, would love to get your feedback on\nhttps://github.com/kaihendry/graphsloc\n\nKind regards from Petaling Jaya,\n"},{"id":"280693","messageId":"20160314023545.GA19753@sigill.intra.peff.net","threadId":"41679","inReplyTo":"1457780426.2632189.547083938.25305E83@webmail.messagingengine.com","subject":"Re: Graph sloc tool for git repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-03-14T02:35:45Z","receivedAt":"2016-03-14T02:35:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 12, 2016 at 07:00:26PM +0800, Kai Hendry wrote:\n\n> I penned a script to plot SLOC of a git project using GNUplot & I\n> thought the fastest way to count code fluctuations was via `git show\n> --numstat`.\n> \n> However that requires some awk counting of the lines:\n> https://github.com/kaihendry/graphsloc/blob/5f31e388e9b655e1801f13885f4311d221663a19/collect-stats.sh#L32\n> \n> Is there a better way I missed? I think there is bug since my graph was\n> a factor of 10 out whilst graphing Linux:\n> https://twitter.com/kaihendry/status/706627679924174848\n\nI think you'll always need to post-process the --numstat output to count\nup lines. But that's fairly minor.\n\nThe bigger problem, I think is in how you handle merges. Imagine I have\ntwo branches, each of which touch the same code:\n\n  git init\n  echo base >file\n  git add file\n  git commit -m base\n\n  echo master >>file\n  git commit -am master\n\n  git checkout -b side HEAD^\n  echo side >>file\n  git commit -am side\n\nand now I merge them, resolving the conflict favor of one side:\n\n  git merge master\n  { echo base; echo master; } >file\n  git commit -am resolved\n\nWhat does --numstat say?\n\n  $ git log --cc --oneline --numstat\n  989c6f7 resolved\n  1       1       file\n  b9bbaf9 side\n  1       0       file\n  087b294 master\n  1       0       file\n  09037ef base\n  1       0       file\n\nIf we add these up, it looks like 3 lines were added. But the end result\nhas only 2 lines! We double-counted the additions on the two branches,\neven though one stomped on the other. And then the merge resolution\nlooks neutral (one line gone, one line added), even though it was where\nwe did the stomping.\n\nTo be honest, I am not sure _what_ the \"--cc --numstat\" is showing\nthere (I added --cc because that is used by \"git show\", which is what\nyour script uses). The actual \"--cc\" (and \"-c\") patches show nothing,\nwhich is right. It kind of looks like we are just showing the diffstat\nagainst the first parent. I'm not sure anyone ever designed what a\n\"combined\" diffstat would look like.\n\nLet's redo our merge and resolve in favor of \"side\":\n\n  git reset --hard HEAD^\n  git merge master\n  { echo base; echo side; } >file\n  git commit -am resolved\n\nNow the numstat is blank. I think it _is_ just showing the first-parent\ndiffstat. So that means each merge is introducing errors into your\ncount, and you'll drift away from accurate.\n\nAnother, related problem, is that you are adding up numbers along\nmultiple simultaneous branches. So going back to my output above, at the\npoint we read the \"master\" commit, we might say that the total\nsloc-count is 2.  That's correct. And then we read \"side\", and say there\nare 3 lines. But that's not right. It doesn't build on master, so we\nstill have only 2 lines. So even _if_ we merged them and the end result\nhad 3 lines (or if we somehow accounted for the double-counting when\nexamining the merge), you'd have inaccuracies through your dataset.\n\nAnother way of thinking about it is that your graph wants to represent a\nsingle linear history, with the sloc-count changing as time moves to the\nright. But that's not what really happened; at any given time, there are\n_several_ sloc-counts, depending on which branch you're following.\n\nBut that can also give us a clue about one solution[1].  For your\npurposes, you don't care about hitting every commit. You just want a\nbunch of linear samples of the form [timestamp, sloc] to feed to\ngnuplot. We can use \"--first-parent\" to walk _a_ linear history and see\na strict progression. And then use \"-m\" to tell git to just show merges\nas the diff against that first parent (i.e., summarizing everything that\nhappened along the side-branch we are not following).\n\nLike (this is back on the \"we resolved as master\" version of my example,\nto illustrate how the merge is shown):\n\n  $ git log --first-parent -m --numstat --oneline\n  4244c8a resolved\n  1       1       file\n  b9bbaf9 side\n  1       0       file\n  09037ef base\n  1       0       file\n\nAnd that count is right. We had one line in our base, one on the\n\"side\" commit, and then the merge didn't change our sloc-count at all\n(we dropped the \"side\" line in favor of \"master\").\n\nFinally, I'll note one other thing in my examples above. Note that I\nused a single \"git log\" invocation, whereas your script reads \"rev-list\"\noutput to start a series of N \"git show\" invocations. I'll bet that took\nquite a long time to run on the kernel. :)  Doing it all in one git-log\nmeans you avoid the process startup overhead, and I'd expect it to run\n~50 times as fast (at least that was what I saw from a quick experiment\non a much smaller repo).\n\n-Peff\n\n[1] The other solution I thought of was to actually count the SLOC of\n    each tree at each commit, rather than worrying about diffs. You can\n    make it efficient by caching the SLOC of blob and tree sha1s, so\n    you don't count the same files over and over. That gives you\n    accurate SLOC-counts at any given point in time, but doesn't address\n    the multiple-branches thing (so you'd see your SLOC bounce up and\n    down in time as you saw commits for various branches).\n"},{"id":"280694","messageId":"20160314025052.GB19753@sigill.intra.peff.net","threadId":"41679","inReplyTo":"20160314023545.GA19753@sigill.intra.peff.net","subject":"Re: Graph sloc tool for git repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-03-14T02:50:53Z","receivedAt":"2016-03-14T02:50:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Mar 13, 2016 at 10:35:45PM -0400, Jeff King wrote:\n\n> Like (this is back on the \"we resolved as master\" version of my example,\n> to illustrate how the merge is shown):\n> \n>   $ git log --first-parent -m --numstat --oneline\n>   4244c8a resolved\n>   1       1       file\n>   b9bbaf9 side\n>   1       0       file\n>   09037ef base\n>   1       0       file\n\nYou'd probably want --reverse, of course, since the point is to build up\nthe count in the same order as time flows.\n\nSo this is the working version I came up with:\n\n    git log --reverse --first-parent -m --format=%ct --numstat |\n    perl -lne '\n      if (/^\\d+$/) {\n              if (defined $time) {\n                      print \"$time $total\"\n              }\n              $time = $&;\n      } elsif (/^(\\d+)\\s+(\\d+)\\s/) {\n              $total += $1 - $2;\n      }\n      END {\n        # flush last entry\n        print \"$time $total\";\n      }\n    '\n\nFor my git.git repo, the final line it produces is:\n\n    1457666843 789457\n\nwhich should be the final sloc-count right now.  If I count the lines in\nthe lines in HEAD, it's close but not quite the same:\n\n   $ git ls-tree -r HEAD |\n     awk '{print $3}' |\n     xargs -n1 git cat-file blob |\n     wc -l\n   790437\n\nI'd guess that the difference comes from a few files which are treated\nas binary (and thus get \"-\" in their numstat output, but happen to have\nnewlines which cause \"wc\" to increment its count).\n\n-Peff\n"}]}