{"thread":{"id":"42889","subject":"How to generate feature branch statistics?","startedAt":"2016-07-20T08:05:50Z","lastAt":"2016-07-20T23:31:29Z","messageCount":7,"participants":["Ernesto Maserati","Jeff King","Jakub Narębski","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"291790","messageId":"CAOHAwykGkfY7M30jT8t0k6Gsdy5QSBHmAPiWYoKibjUgS-G6hg@mail.gmail.com","threadId":"42889","inReplyTo":null,"subject":"How to generate feature branch statistics?","fromName":"Ernesto Maserati","fromEmail":"ernesto.2.maserati@gmail.com","sentAt":"2016-07-20T08:05:09Z","receivedAt":"2016-07-20T08:05:50Z","isPatch":false,"sender":{"key":"ernesto.2.maserati@gmail.com","avatar":null},"body":"I assume that feature branches are not frequently enough merged into\nmaster. Because of that we discover bugs later than we could with a more\ncontinuous code integration. I don't want to discuss here whether feature\nbranches are good or bad.\n\nI want just to ask is there a way how to generate a statistic for the\naverage duration of feature branches until they are merged to the master? I\nwould like to know if it is 1 day, 2 days or lets say 8 or 17 days. Also it\nwould be interesting to see the statistical outliers.\n\nI hope my motivation became clear and what kind of git repository data I\nwould like to produce.\n\nAny ideas?\n"},{"id":"291803","messageId":"20160720131424.GC17469@sigill.intra.peff.net","threadId":"42889","inReplyTo":"CAOHAwykGkfY7M30jT8t0k6Gsdy5QSBHmAPiWYoKibjUgS-G6hg@mail.gmail.com","subject":"Re: How to generate feature branch statistics?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-20T13:14:24Z","receivedAt":"2016-07-20T13:14:49Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 20, 2016 at 10:05:09AM +0200, Ernesto Maserati wrote:\n\n> I assume that feature branches are not frequently enough merged into\n> master. Because of that we discover bugs later than we could with a more\n> continuous code integration. I don't want to discuss here whether feature\n> branches are good or bad.\n> \n> I want just to ask is there a way how to generate a statistic for the\n> average duration of feature branches until they are merged to the master? I\n> would like to know if it is 1 day, 2 days or lets say 8 or 17 days. Also it\n> would be interesting to see the statistical outliers.\n\nIn a workflow that merges feature branches to master, you can generally\nrecognize them by looking for merges along the first-parent chain of\ncommits:\n\n  git log --first-parent --merges master\n\n(Depending on your workflow, some feature branches may be fast-forwards\nwith no merge commit, so this is just a sampling. Some workflows use\n\"git merge --no-ff\" to merge in feature branches, so this would see all\nof them).\n\nAnd then for each merge, you can get the set of commits that were merged\nin (it is the commits in the second parent that are not in the first).\nThe bottom-most one is the \"start\" of the branch (or close to it; of\ncourse the author started writing code before they made a commit), and\nthe \"end\" is the merge itself.\n\nSo something like:\n\n  git rev-list --first-parent --merges master |\n  while read merge; do\n\tstart=$(git log --format=%at $merge^1..$merge^2 | tail -1)\n\tend=$(git log -1 --format=%at $merge)\n\tsubject=$(git log -1 --format=%s $merge)\n\techo \"$((end - start)) $subject\"\n  done\n\nThat should output a sequence of topic branch merges prefixed by the\nnumber of seconds they were active. Two exercises for the reader:\n\n  1. Converting seconds into some more useful time scale. :)\n\n  2. This can probably be done with fewer invocations of git,\n     which would be more efficient.\n\n-Peff\n"},{"id":"291810","messageId":"578F8306.3070306@gmail.com","threadId":"42889","inReplyTo":"CAOHAwykGkfY7M30jT8t0k6Gsdy5QSBHmAPiWYoKibjUgS-G6hg@mail.gmail.com","subject":"Re: How to generate feature branch statistics?","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-20T13:56:22Z","receivedAt":"2016-07-20T13:56:33Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-20 o 10:05, Ernesto Maserati pisze:\n\n> I assume that feature branches are not frequently enough merged into\n> master. Because of that we discover bugs later than we could with a more\n> continuous code integration. I don't want to discuss here whether feature\n> branches are good or bad.\n> \n> I want just to ask is there a way how to generate a statistic for the\n> average duration of feature branches until they are merged to the master? I\n> would like to know if it is 1 day, 2 days or lets say 8 or 17 days. Also it\n> would be interesting to see the statistical outliers.\n> \n> I hope my motivation became clear and what kind of git repository data I\n> would like to produce.\n> \n> Any ideas?\n\nThere are at least two tools to generate statistics about git repository,\nnamely Gitstat (https://sourceforge.net/projects/gitstat) and GitStats\n(https://github.com/hoxu/gitstats), both generating repo statistics as\na web page. You can probably find more... but I don't know if any includes\nthe statistics you need.\n\nI assume that you have some way of determining if the merge in 'master'\nbranch is a merge of a topic branch, or of long-lived graduation branch\n(e.g. 'maint' or equivalent). To simplify the situation, I assume that\nthe only merges in master are merges of topic branches:\n\n  git rev-list --min-parents=2 master | \n  while read merge_rev; do \n\nYou might want to add \"--grep=maint --invert-grep\" or something like\nthat to exclude merges of 'maint' branch.\n\t\nWe can get date of merge (authordate with %ad/%at, or committerdate\nwith %cd/%ct), as an epoch (seconds since 1970 -- which is good for\ncomparing datetimes and getting the interval between two events)\n\n     MERGE_DATE=$(git show -s --date=format:%s --pretty=%ad $merge_rev)\n\nAssuming that topic branches are always merged using two-head merge\nas a second parent (--first-parent ancestry for master in master branch\nonly), then we can get the first revision on a merged topic branch with\n\n     FIRST_REV=$(git rev-list $merge_rev^2 ^$merge_rev^1 | tail -1)\n\nWe can extract the date from this revision in the same way\n\n     FIRST_DATE=$(git show -s --pretty=%at $FIRST_REV)\n\nPrint the difference (here to standard output, you might want to write\nto a file)\n\n     echo $(expr $MERGE_DATE - $FIRST_DATE)\n\nAnd finish the loop.\n\n  done\n\nThen pass the output to some histogramming or statistics tool... or use\na spreadsheet. Note the results are in seconds.\n\nHTH (not checked much)\n-- \nJakub Narębski\n"},{"id":"291846","messageId":"578FBEAE.9080307@gmail.com","threadId":"42889","inReplyTo":"578F8306.3070306@gmail.com","subject":"Re: How to generate feature branch statistics?","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-20T18:10:54Z","receivedAt":"2016-07-20T18:11:14Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-20 o 15:56, Jakub Narębski pisze:\n> W dniu 2016-07-20 o 10:05, Ernesto Maserati pisze:\n> \n>> I assume that feature branches are not frequently enough merged into\n>> master. Because of that we discover bugs later than we could with a more\n>> continuous code integration. I don't want to discuss here whether feature\n>> branches are good or bad.\n>>\n>> I want just to ask is there a way how to generate a statistic for the\n>> average duration of feature branches until they are merged to the master? I\n>> would like to know if it is 1 day, 2 days or lets say 8 or 17 days. Also it\n>> would be interesting to see the statistical outliers.\n>>\n>> I hope my motivation became clear and what kind of git repository data I\n>> would like to produce.\n>>\n>> Any ideas?\n> \n> There are at least two tools to generate statistics about git repository,\n> namely Gitstat (https://sourceforge.net/projects/gitstat) and GitStats\n> (https://github.com/hoxu/gitstats), both generating repo statistics as\n> a web page. You can probably find more... but I don't know if any includes\n> the statistics you need.\n> \n> I assume that you have some way of determining if the merge in 'master'\n> branch is a merge of a topic branch, or of long-lived graduation branch\n> (e.g. 'maint' or equivalent). To simplify the situation, I assume that\n> the only merges in master are merges of topic branches:\n> \n>   git rev-list --min-parents=2 master | \n\nSelf correction: Here you need to use --first-parent, as in Peff answer\n(which also uses less git invocations, and less of git porcelain).\n\nI wonder if it is something that libgit2 would be helpful...\n-- \nJakub Narębski\n\n"},{"id":"291848","messageId":"xmqqd1m8du1b.fsf@gitster.mtv.corp.google.com","threadId":"42889","inReplyTo":"20160720131424.GC17469@sigill.intra.peff.net","subject":"Re: How to generate feature branch statistics?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-20T18:49:20Z","receivedAt":"2016-07-20T18:49:28Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> In a workflow that merges feature branches to master, you can generally\n> recognize them by looking for merges along the first-parent chain of\n> commits:\n>\n>   git log --first-parent --merges master\n>\n> (Depending on your workflow, some feature branches may be fast-forwards\n> with no merge commit, so this is just a sampling. Some workflows use\n> \"git merge --no-ff\" to merge in feature branches, so this would see all\n> of them).\n> And then for each merge, you can get the set of commits that were merged\n> in (it is the commits in the second parent that are not in the first).\n> The bottom-most one is the \"start\" of the branch (or close to it; of\n> course the author started writing code before they made a commit), and\n> the \"end\" is the merge itself.\n\nA few things to keep in mind are\n\n * A feature branch may be merged to the master multiple times,\n   when the feature branch is properly managed.\n\n   E.g. It may have been once thought to be complete with 3 commits,\n   get merged to 'master', then a bug is discovered and gain its\n   fourth commits to fix the bug and merged to 'master' again,\n   resulting in a topology like this:\n\n         A---B---C-----------D (feature)\n        /         \\           \\\n    ---o---o---o---1---o---o---2---o (master)\n\n   \"git log --first-parent --merges master\" will first find commit\n   '2' that merged the feature for the second time, bringing in\n   commit 'D', and then it will find commit '1' that merged the\n   feature previously, bringing in commit 'A', 'B' and 'C'.\n\n * A feature branch that depends on other feature may have merges on\n   their own.  You may start a feature X that depends on another\n   features Y and Z that are not yet in 'master', in addition to\n   depending on things in 'master' that have been added since Y and\n   Z forked from it.  In such a case, your feature X may look like\n   this:\n\n                 .-------------------1----------2--------x---x (feature X)\n                /                   /          /\n       y---y---y (feature Y)       /          /\n      /                           /          /\n  ---o---o---o---o---o---o---o---0 (master) /\n          \\                                /\n           z---z (feature Z)              /\n                \\                        /\n                 .----------------------.\n\n   where '1' and '2' are merges of feature Y and then Z into the tip\n   of 'master' when you start working on feature X.\n\n   And then feature Y and feature Z may graduate to 'master' before\n   your feature X is ready to do so, resulting in something like:\n\n                 .-------------------1----------2--------x---x (feature X)\n                /                   /          /\n       y---y---y (feature Y) ----  / -------  /  --.\n      /                           /          /      \\\n  ---o---o---o---o---o---o---o---o---o---o---o---o---Y---Z (master)\n          \\                                /            /\n           z---z (feature Z) ----------   /  ----------.\n                \\                        /\n                 .----------------------.\n\n   where 'Y' and 'Z' are merges of features Y and Z to 'master'.\n   After that, feature X may become ready to be merged, resulting in:\n\n                 .-------------------1----------2--------x---x (feature X)\n                /                   /          /              \\    \n       y---y---y (feature Y) ----  / -------  /  --.           \\\n      /                           /          /      \\           \\\n  ---o---o---o---o---o---o---o---o---o---o---o---o---Y---Z---o---X (master)\n          \\                                /            /\n           z---z (feature Z) ----------   /  ----------.\n                \\                        /\n                 .----------------------.\n\n  When \"git log --first-parent --merges master\" finds X, it would\n  notice that it pulled in commits '1', '2' and two 'x'.  The \"tool\"\n  to inspect the history needs to be careful deciding if '1' and '2'\n  are the part of feature X.  There are variants that make it tricky\n  (e.g. 'Y' may not have yet been merged to 'master' when 'X' is\n  merged, in which case you may end up pulling both 'x' and 'y' into\n  'master' with a single merge), which should be avoided if feature\n  branches are managed carefully, but not everybody is careful when\n  managing their history.\n\nComing back to the introduction of the original message:\n\n>> I assume that feature branches are not frequently enough merged into\n>> master. Because of that we discover bugs later than we could with a more\n>> continuous code integration. I don't want to discuss here whether feature\n>> branches are good or bad.\n\nFor our own history and workflow, the duration between the inception\nof a topic branch and the time it gets merged to 'master' is not all\nthat interesting.  More interesting numbers are:\n\n * The duration between the time a topic hits 'next' and the time it\n   gets merged to 'master'.  This is the time the developers and\n   testers are using the new feature in their own work to make sure\n   it does not have any ill effect.\n\n * The percetage of topics that is merged to 'master' with some\n   follow-up changes since it hits 'next'.  This is an approximate\n   for the number of bugs that are caught by developers and testers\n   before a new feature goes to the general public.\n"},{"id":"291893","messageId":"579004CC.8090503@gmail.com","threadId":"42889","inReplyTo":"xmqqd1m8du1b.fsf@gitster.mtv.corp.google.com","subject":"Re: How to generate feature branch statistics?","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2016-07-20T23:10:04Z","receivedAt":"2016-07-20T23:10:18Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"W dniu 2016-07-20 o 20:49, Junio C Hamano pisze:\n\n> For our own history and workflow, the duration between the inception\n> of a topic branch and the time it gets merged to 'master' is not all\n> that interesting.\n\nNb. if I haven't messed something up (the git history is not simple\nmerging of topic branches into mainline), the shortest time from\ncreating a branch to merging it in git.git is 7 seconds (probably\nit was a bugfix-type of a topic branch), the longest if I did it\ncorrectly is slightly less than 4 years (???): 641830c.\n\n-- \nJakub Narębski\n\n"},{"id":"291895","messageId":"CAPc5daUjK1k8teTw8enb3yNwo+WesJY3BVHM34Wdqu31czUB4Q@mail.gmail.com","threadId":"42889","inReplyTo":"579004CC.8090503@gmail.com","subject":"Re: How to generate feature branch statistics?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-20T23:31:04Z","receivedAt":"2016-07-20T23:31:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Wed, Jul 20, 2016 at 4:10 PM, Jakub Narębski <jnareb@gmail.com> wrote:\n> W dniu 2016-07-20 o 20:49, Junio C Hamano pisze:\n>\n>> For our own history and workflow, the duration between the inception\n>> of a topic branch and the time it gets merged to 'master' is not all\n>> that interesting.\n>\n> Nb. if I haven't messed something up (the git history is not simple\n> merging of topic branches into mainline), the shortest time from\n> creating a branch to merging it in git.git is 7 seconds (probably\n> it was a bugfix-type of a topic branch), the longest if I did it\n> correctly is slightly less than 4 years (???): 641830c.\n\nThe former is quite understandable. The point of having such a topic\nis so that it can be merged down to older maintenance releases.\n"}]}