{"thread":{"id":"21538","subject":"[RFC/PATCH] Documentation: add \"Fighting regressions with git bisect\" article","startedAt":"2009-11-08T15:09:47Z","lastAt":"2009-11-08T15:09:47Z","messageCount":1,"participants":["Christian Couder"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"127086","messageId":"20091108150948.12334.12145.chriscool@tuxfamily.org","threadId":"21538","inReplyTo":null,"subject":"[RFC/PATCH] Documentation: add \"Fighting regressions with git bisect\" article","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2009-11-08T15:09:47Z","receivedAt":"2009-11-08T15:09:47Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This patch adds an asciidoc version of the \"Fighting regressions with\ngit bisect\" article that the author wrote for the Linux-Kongress\n2009 (http://www.linux-kongress.org/2009).\n\nThis paper might be interesting to people who want to learn as much as\npossible about \"git bisect\" from a single document.\n\nThe slides of the related presentation are available at:\n\nhttp://www.linux-kongress.org/2009/slides/fighting_regressions_with_git_bisect_christian_couder.pdf\n\nBut the Linux Kongress people will not publish this paper online because\nthey print the papers on their UpTimes magazine\n(http://www.lob.de/isbn/978-3-86541-358-1). But they don't take away the\nrights of the author (which is very nice), so I have the right to publish\nit.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n Documentation/Makefile              |    1 +\n Documentation/git-bisect-lk2009.txt | 1359 +++++++++++++++++++++++++++++++++++\n Documentation/git-bisect.txt        |    5 +\n 3 files changed, 1365 insertions(+), 0 deletions(-)\n create mode 100644 Documentation/git-bisect-lk2009.txt\n\n\tHi,\n\n\tHere is the paper that I wrote for the Linux Kongress 2009 in Dresden.\n\tYou can see the PDF version here:\n\n\thttp://blog.couder.net/public/lk2009-git_bisect.pdf\n\n\tAs I wanted to have an HTML version of it, I reworked it a little to\n\tconvert it to asciidoc. (In fact I wrote it nearly in asciidoc in the\n\tfirst place, but I had to submit it in other formats.)\n\n\tIt is not only a technical paper, and not only a tutorial either so\n\tI am not sure where it would be better to add it.\n\n\tThanks again to Junio and Ingo for their help and comments on it,\n\tChristian.\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex cd5b439..3f59952 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -17,6 +17,7 @@ DOC_HTML=$(MAN_HTML)\n ARTICLES = howto-index\n ARTICLES += everyday\n ARTICLES += git-tools\n+ARTICLES += git-bisect-lk2009\n # with their own formatting rules.\n SP_ARTICLES = howto/revert-branch-rebase howto/using-merge-subtree user-manual\n API_DOCS = $(patsubst %.txt,%,$(filter-out technical/api-index-skel.txt technical/api-index.txt, $(wildcard technical/api-*.txt)))\ndiff --git a/Documentation/git-bisect-lk2009.txt b/Documentation/git-bisect-lk2009.txt\nnew file mode 100644\nindex 0000000..ca30078\n--- /dev/null\n+++ b/Documentation/git-bisect-lk2009.txt\n@@ -0,0 +1,1359 @@\n+Fighting regressions with git bisect\n+====================================\n+:Author: Christian Couder\n+:Email: chriscool@tuxfamily.org\n+:Date: 2009/11/08\n+\n+Abstract\n+--------\n+\n+\"git bisect\" enables software users and developers to easily find the\n+commit that introduced a regression. We show why it is important to\n+have good tools to fight regressions. We describe how \"git bisect\"\n+works from the outside and the algorithms it uses inside. Then we\n+explain how to take advantage of \"git bisect\" to improve current\n+practices. And we discuss how \"git bisect\" could improve in the\n+future.\n+\n+\n+Introduction to \"git bisect\"\n+----------------------------\n+\n+Git is a Distributed Version Control system (DVCS) created by Linus\n+Torvalds and maintained by Junio Hamano.\n+\n+In Git like in many other Version Control Systems (VCS), the different\n+states of the data that is managed by the system are called\n+commits. And, as VCS are mostly used to manage software source code,\n+sometimes \"interesting\" changes of behavior in the software are\n+introduced in some commits.\n+\n+In fact people are specially interested in commits that introduce a\n+\"bad\" behavior, called a bug or a regression. They are interested in\n+these commits because a commit (hopefully) contains a very small set\n+of source code changes. And it's much easier to understand and\n+properly fix a problem when you only need to check a very small set of\n+changes, than when you don't know where look in the first place.\n+\n+So to help people find commits that introduce a \"bad\" behavior, the\n+\"git bisect\" set of commands was invented. And it follows of course\n+that in \"git bisect\" parlance, commits where the \"interesting\n+behavior\" is present are called \"bad\" commits, while other commits are\n+called \"good\" commits. And a commit that introduce the behavior we are\n+interested in is called a \"first bad commit\". Note that there could be\n+more than one \"first bad commit\" in the commit space we are searching.\n+\n+So \"git bisect\" is designed to help find a \"first bad commit\". And to\n+be as efficient as possible, it tries to perform a binary search.\n+\n+\n+Fighting regressions overview\n+-----------------------------\n+\n+Regressions: a big problem\n+~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Regressions are a big problem in the software industry. But it's\n+difficult to put some real numbers behind that claim.\n+\n+There are some numbers about bugs in general, like a NIST study in\n+2002 <<1>> that said:\n+\n+_____________\n+Software bugs, or errors, are so prevalent and so detrimental that\n+they cost the U.S. economy an estimated $59.5 billion annually, or\n+about 0.6 percent of the gross domestic product, according to a newly\n+released study commissioned by the Department of Commerce's National\n+Institute of Standards and Technology (NIST). At the national level,\n+over half of the costs are borne by software users and the remainder\n+by software developers/vendors.  The study also found that, although\n+all errors cannot be removed, more than a third of these costs, or an\n+estimated $22.2 billion, could be eliminated by an improved testing\n+infrastructure that enables earlier and more effective identification\n+and removal of software defects. These are the savings associated with\n+finding an increased percentage (but not 100 percent) of errors closer\n+to the development stages in which they are introduced. Currently,\n+over half of all errors are not found until \"downstream\" in the\n+development process or during post-sale software use.\n+_____________\n+\n+And then:\n+\n+_____________\n+Software developers already spend approximately 80 percent of\n+development costs on identifying and correcting defects, and yet few\n+products of any type other than software are shipped with such high\n+levels of errors.\n+_____________\n+\n+Eventually the conclusion started with:\n+\n+_____________\n+The path to higher software quality is significantly improved software\n+testing.\n+_____________\n+\n+There are other estimates saying that 80% of the cost related to\n+software is about maintenance <<2>>.\n+\n+Though, according to Wikipedia <<3>>:\n+\n+_____________\n+A common perception of maintenance is that it is merely fixing\n+bugs. However, studies and surveys over the years have indicated that\n+the majority, over 80%, of the maintenance effort is used for\n+non-corrective actions (Pigosky 1997). This perception is perpetuated\n+by users submitting problem reports that in reality are functionality\n+enhancements to the system.\n+_____________\n+\n+But we can guess that improving on existing software is very costly\n+because you have to watch out for regressions. At least this would\n+make the above studies consistent among themselves.\n+\n+Of course some kind of software is developed, then used during some\n+time without being improved on much, and then finally thrown away. In\n+this case, of course, regressions may not be a big problem. But on the\n+other hand, there is a lot of big software that is continually\n+developed and maintained during years or even tens of years by a lot\n+of people. And as there are often many people who depend (sometimes\n+critically) on such software, regressions are a really big problem.\n+\n+One such software is the linux kernel. And if we look at the linux\n+kernel, we can see that a lot of time and effort is spent to fight\n+regressions. The release cycle start with a 2 weeks long merge\n+window. Then the first release candidate (rc) version is tagged. And\n+after that about 7 or 8 more rc versions will appear with around one\n+week between each of them, before the final release.\n+\n+The time between the first rc release and the final release is\n+supposed to be used to test rc versions and fight bugs and especially\n+regressions. And this time is more than 80% of the release cycle\n+time. But this is not the end of the fight yet, as of course it\n+continues after the release.\n+\n+And then this is what Ingo Molnar (a well known linux kernel\n+developer) says about his use of git bisect:\n+\n+_____________\n+I most actively use it during the merge window (when a lot of trees\n+get merged upstream and when the influx of bugs is the highest) - and\n+yes, there have been cases that i used it multiple times a day. My\n+average is roughly once a day.\n+_____________\n+\n+So regressions are fought all the time by developers, and indeed it is\n+well known that bugs should be fixed as soon as possible, so as soon\n+as they are found. That's why it is interesting to have good tools for\n+this purpose.\n+\n+Other tools to fight regressions\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+So what are the tools used to fight regressions? They are nearly the\n+same as those used to fight regular bugs. The only specific tools are\n+test suites and tools similar as \"git bisect\".\n+\n+Test suites are very nice. But when they are used alone, they are\n+supposed to be used so that all the tests are checked after each\n+commit. This means that they are not very efficient, because many\n+tests are run for no interesting result, and they suffer from\n+combinational explosion.\n+\n+In fact the problem is that big software often has many different\n+configuration options and that each test case should pass for each\n+configuration after each commit. So if you have for each release: N\n+configurations, M commits and T test cases, you should perform:\n+\n+-------------\n+N * M * T tests\n+-------------\n+\n+where N, M and T are all growing with the size your software.\n+\n+So very soon it will not be possible to completely test everything.\n+\n+And if some bugs slip through your test suite, then you can add a test\n+to your test suite. But if you want to use your new improved test\n+suite to find where the bug slipped in, then you will either have to\n+emulate a bisection process or you will perhaps bluntly test each\n+commit backward starting from the \"bad\" commit you have which may be\n+very wasteful.\n+\n+\"git bisect\" overview\n+---------------------\n+\n+Starting a bisection\n+~~~~~~~~~~~~~~~~~~~~\n+\n+The first \"git bisect\" subcommand to use is \"git bisect start\" to\n+start the search. Then bounds must be set to limit the commit\n+space. This is done usually by giving one \"bad\" and at least one\n+\"good\" commit. They can be passed in the initial call to \"git bisect\n+start\" like this:\n+\n+-------------\n+$ git bisect start [BAD [GOOD...]]\n+-------------\n+\n+or they can be set using:\n+\n+-------------\n+$ git bisect bad [COMMIT]\n+-------------\n+\n+and:\n+\n+-------------\n+$ git bisect good [COMMIT...]\n+-------------\n+\n+where BAD, GOOD and COMMIT are all names that can be resolved to a\n+commit.\n+\n+Then \"git bisect\" will checkout a commit of its choosing and ask the\n+user to test it, like this:\n+\n+-------------\n+$ git bisect start v2.6.27 v2.6.25\n+Bisecting: 10928 revisions left to test after this (roughly 14 steps)\n+[2ec65f8b89ea003c27ff7723525a2ee335a2b393] x86: clean up using max_low_pfn on 32-bit\n+-------------\n+\n+Note that the example that we will use is really a toy example, we\n+will be looking for the first commit that has a version like\n+\"2.6.26-something\", that is the commit that has a \"SUBLEVEL = 26\" line\n+in the top level Makefile. This is a toy example because there are\n+better ways to find this commit with git than using \"git bisect\" (for\n+example \"git blame\" or \"git log -S<string>\").\n+\n+Driving a bisection manually\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+At this point there are basically 2 ways to drive the search. It can\n+be driven manually by the user or it can be driven automatically by a\n+script or a command.\n+\n+If the user is driving it, then at each step of the search, the user\n+will have to test the current commit and say if it is \"good\" or \"bad\"\n+using the \"git bisect good\" or \"git bisect bad\" commands respectively\n+that have been described above. For example:\n+\n+-------------\n+$ git bisect bad\n+Bisecting: 5480 revisions left to test after this (roughly 13 steps)\n+[66c0b394f08fd89236515c1c84485ea712a157be] KVM: kill file->f_count abuse in kvm\n+-------------\n+\n+And after a few more steps like that, \"git bisect\" will eventually\n+find a first bad commit:\n+\n+-------------\n+$ git bisect bad\n+2ddcca36c8bcfa251724fe342c8327451988be0d is the first bad commit\n+commit 2ddcca36c8bcfa251724fe342c8327451988be0d\n+Author: Linus Torvalds <torvalds@linux-foundation.org>\n+Date:   Sat May 3 11:59:44 2008 -0700\n+\n+    Linux 2.6.26-rc1\n+\n+:100644 100644 5cf8258195331a4dbdddff08b8d68642638eea57 4492984efc09ab72ff6219a7bc21fb6a957c4cd5 M      Makefile\n+-------------\n+\n+At this point we can see what the commit does, check it out (if it's\n+not already checked out) or tinker with it, for example:\n+\n+-------------\n+$ git show HEAD\n+commit 2ddcca36c8bcfa251724fe342c8327451988be0d\n+Author: Linus Torvalds <torvalds@linux-foundation.org>\n+Date:   Sat May 3 11:59:44 2008 -0700\n+\n+    Linux 2.6.26-rc1\n+\n+diff --git a/Makefile b/Makefile\n+index 5cf8258..4492984 100644\n+--- a/Makefile\n++++ b/Makefile\n+@@ -1,7 +1,7 @@\n+ VERSION = 2\n+ PATCHLEVEL = 6\n+-SUBLEVEL = 25\n+-EXTRAVERSION =\n++SUBLEVEL = 26\n++EXTRAVERSION = -rc1\n+ NAME = Funky Weasel is Jiggy wit it\n+\n+ # *DOCUMENTATION*\n+-------------\n+\n+And when we are finished we can use \"git bisect reset\" to go back to\n+the branch we were in before we started bisecting:\n+\n+-------------\n+$ git bisect reset\n+Checking out files: 100% (21549/21549), done.\n+Previous HEAD position was 2ddcca3... Linux 2.6.26-rc1\n+Switched to branch 'master'\n+-------------\n+\n+Driving a bisection automatically\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The other way to drive the bisection process is to tell \"git bisect\"\n+to launch a script or command at each bisection step to know if the\n+current commit is \"good\" or \"bad\". To do that, we use the \"git bisect\n+run\" command. For example:\n+\n+-------------\n+$ git bisect start v2.6.27 v2.6.25\n+Bisecting: 10928 revisions left to test after this (roughly 14 steps)\n+[2ec65f8b89ea003c27ff7723525a2ee335a2b393] x86: clean up using max_low_pfn on 32-bit\n+$\n+$ git bisect run grep '^SUBLEVEL = 25' Makefile\n+running grep ^SUBLEVEL = 25 Makefile\n+Bisecting: 5480 revisions left to test after this (roughly 13 steps)\n+[66c0b394f08fd89236515c1c84485ea712a157be] KVM: kill file->f_count abuse in kvm\n+running grep ^SUBLEVEL = 25 Makefile\n+SUBLEVEL = 25\n+Bisecting: 2740 revisions left to test after this (roughly 12 steps)\n+[671294719628f1671faefd4882764886f8ad08cb] V4L/DVB(7879): Adding cx18 Support for mxl5005s\n+...\n+...\n+running grep ^SUBLEVEL = 25 Makefile\n+Bisecting: 0 revisions left to test after this (roughly 0 steps)\n+[2ddcca36c8bcfa251724fe342c8327451988be0d] Linux 2.6.26-rc1\n+running grep ^SUBLEVEL = 25 Makefile\n+2ddcca36c8bcfa251724fe342c8327451988be0d is the first bad commit\n+commit 2ddcca36c8bcfa251724fe342c8327451988be0d\n+Author: Linus Torvalds <torvalds@linux-foundation.org>\n+Date:   Sat May 3 11:59:44 2008 -0700\n+\n+    Linux 2.6.26-rc1\n+\n+:100644 100644 5cf8258195331a4dbdddff08b8d68642638eea57 4492984efc09ab72ff6219a7bc21fb6a957c4cd5 M      Makefile\n+bisect run success\n+-------------\n+\n+In this example, we passed \"grep '^SUBLEVEL = 25' Makefile\" as\n+parameter to \"git bisect run\". This means that at each step, the grep\n+command we passed will be launched. And if it exits with code 0 (that\n+means success) then git bisect will mark the current state as\n+\"good\". If it exits with code 1 (or any code between 1 and 127\n+included, except the special code 125), then the current state will be\n+marked as \"bad\".\n+\n+Exit code between 128 and 255 are special to \"git bisect run\". They\n+make it stop immediately the bisection process. This is useful for\n+example if the command passed takes too long to complete, because you\n+can kill it with a signal and it will stop the bisection process.\n+\n+It can also be useful in scripts passed to \"git bisect run\" to \"exit\n+255\" if some very abnormal situation is detected.\n+\n+Avoiding untestable commits\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Sometimes it happens that the current state cannot be tested, for\n+example if it does not compile because there was a bug preventing it\n+at that time. This is what the special exit code 125 is for. It tells\n+\"git bisect run\" that the current commit should be marked as\n+untestable and that another one should be chosen and checked out.\n+\n+If the bisection process is driven manually, you can use \"git bisect\n+skip\" to do the same thing. (In fact the special exit code 125 makes\n+\"git bisect run\" use \"git bisect skip\" in the background.)\n+\n+Or if you want more control, you can inspect the current state using\n+for example \"git bisect visualize\". It will launch gitk (or \"git log\"\n+if the DISPLAY environment variable is not set) to help you find a\n+better bisection point.\n+\n+Either way, if you have a string of untestable commits, it might\n+happen that the regression you are looking for has been introduced by\n+one of these untestable commits. In this case it's not possible to\n+tell for sure which commit introduced the regression.\n+\n+So if you used \"git bisect skip\" (or the run script exited with\n+special code 125) you could get a result like this:\n+\n+-------------\n+There are only 'skip'ped commits left to test.\n+The first bad commit could be any of:\n+15722f2fa328eaba97022898a305ffc8172db6b1\n+78e86cf3e850bd755bb71831f42e200626fbd1e0\n+e15b73ad3db9b48d7d1ade32f8cd23a751fe0ace\n+070eab2303024706f2924822bfec8b9847e4ac1b\n+We cannot bisect more!\n+-------------\n+\n+Saving a log and replaying it\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+If you want to show other people your bisection process, you can get a\n+log using for example:\n+\n+-------------\n+$ git bisect log > bisect_log.txt\n+-------------\n+\n+And it is possible to replay it using:\n+\n+-------------\n+$ git bisect replay bisect_log.txt\n+-------------\n+\n+\n+\"git bisect\" details\n+--------------------\n+\n+Bisection algorithm\n+~~~~~~~~~~~~~~~~~~~\n+\n+As the Git commits form a directed acyclic graph (DAG), finding the\n+best bisection commit to test at each step is not so simple. Anyway\n+Linus found and implemented a \"truly stupid\" algorithm, later improved\n+by Junio Hamano, that works quite well.\n+\n+So the algorithm used by \"git bisect\" to find the best bisection\n+commit when there are no skipped commits is the following:\n+\n+1) keep only the commits that:\n+\n+a) are ancestor of the \"bad\" commit (including the \"bad\" commit itself),\n+b) are not ancestor of a \"good\" commit (excluding the \"good\" commits).\n+\n+This means that we get rid of the uninteresting commits in the DAG.\n+\n+For example if we start with a graph like this:\n+\n+-------------\n+G-Y-G-W-W-W-X-X-X-X\n+           \\ /\n+            W-W-B\n+           /\n+Y---G-W---W\n+ \\ /   \\\n+Y-Y     X-X-X-X\n+\n+-> time goes this way ->\n+-------------\n+\n+where B is the \"bad\" commit, \"G\" are \"good\" commits and W, X, and Y\n+are other commits, we will get the following graph after this first\n+step:\n+\n+-------------\n+W-W-W\n+     \\\n+      W-W-B\n+     /\n+W---W\n+-------------\n+\n+So only the W and B commits will be kept. Because commits X and Y will\n+have been removed by rules a) and b) respectively, and because commits\n+G are removed by rule b) too.\n+\n+Note for git users, that it is equivalent as keeping only the commit\n+given by:\n+\n+-------------\n+git rev-list BAD --not GOOD1 GOOD2...\n+-------------\n+\n+Also note that we don't require the commits that are kept to be\n+descendants of a \"good\" commit. So in the following example, commits W\n+and Z will be kept:\n+\n+-------------\n+G-W-W-W-B\n+   /\n+Z-Z\n+-------------\n+\n+2) starting from the \"good\" ends of the graph, associate to each\n+commit the number of ancestors it has plus one\n+\n+For example with the following graph where H is the \"bad\" commit and A\n+and D are some parents of some \"good\" commits:\n+\n+-------------\n+A-B-C\n+     \\\n+      F-G-H\n+     /\n+D---E\n+-------------\n+\n+this will give:\n+\n+-------------\n+1 2 3\n+A-B-C\n+     \\6 7 8\n+      F-G-H\n+1   2/\n+D---E\n+-------------\n+\n+3) associate to each commit: min(X, N - X)\n+\n+where X is the value associated to the commit in step 2) and N is the\n+total number of commits in the graph.\n+\n+In the above example we have N = 8, so this will give:\n+\n+-------------\n+1 2 3\n+A-B-C\n+     \\2 1 0\n+      F-G-H\n+1   2/\n+D---E\n+-------------\n+\n+4) the best bisection point is the commit with the highest associated\n+number\n+\n+So in the above example the best bisection point is commit C.\n+\n+5) note that some shortcuts are implemented to speed up the algorithm\n+\n+As we know N from the beginning, we know that min(X, N - X) can't be\n+greater than N/2. So during steps 2) and 3), if we would associate N/2\n+to a commit, then we know this is the best bisection point. So in this\n+case we can just stop processing any other commit and return the\n+current commit.\n+\n+Bisection algorithm debugging\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+For any commit graph, you can see the number associated with each\n+commit using \"git rev-list --bisect-all\".\n+\n+For example, for the above graph, a command like:\n+\n+-------------\n+$ git rev-list --bisect-all BAD --not GOOD1 GOOD2\n+-------------\n+\n+would output something like:\n+\n+-------------\n+e15b73ad3db9b48d7d1ade32f8cd23a751fe0ace (dist=3)\n+15722f2fa328eaba97022898a305ffc8172db6b1 (dist=2)\n+78e86cf3e850bd755bb71831f42e200626fbd1e0 (dist=2)\n+a1939d9a142de972094af4dde9a544e577ddef0e (dist=2)\n+070eab2303024706f2924822bfec8b9847e4ac1b (dist=1)\n+a3864d4f32a3bf5ed177ddef598490a08760b70d (dist=1)\n+a41baa717dd74f1180abf55e9341bc7a0bb9d556 (dist=1)\n+9e622a6dad403b71c40979743bb9d5be17b16bd6 (dist=0)\n+-------------\n+\n+Bisection algorithm discussed\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+First let's define \"best bisection point\". We will say that a commit X\n+is a best bisection point or a best bisection commit if knowing its\n+state (\"good\" or \"bad\") gives as much information as possible whether\n+the state of the commit happens to be \"good\" or \"bad\".\n+\n+This means that the best bisection commits are the commits where the\n+following function is maximum:\n+\n+-------------\n+f(X) = min(information_if_good(X), information_if_bad(X))\n+-------------\n+\n+where information_if_good(X) is the information we get if X is good\n+and information_if_bad(X) is the information we get if X is bad.\n+\n+Now we will suppose that there is only one \"first bad commit\". This\n+means that all its descendants are \"bad\" and all the other commits are\n+\"good\". And we will suppose that all commits have an equal probability\n+of being good or bad, or of being the first bad commit, so knowing the\n+state of c commits gives always the same amount of information\n+wherever these c commits are on the graph and whatever c is. (So we\n+suppose that these commits being for example on a branch or near a\n+good or a bad commit does not give more or less information).\n+\n+Let's also suppose that we have a cleaned up graph like one after step\n+1) in the bisection algorithm above. This means that we can measure\n+the information we get in terms of number of commit we can remove from\n+the graph..\n+\n+And let's take a commit X in the graph.\n+\n+If X is found to be \"good\", then we know that its ancestors are all\n+\"good\", so we want to say that:\n+\n+-------------\n+information_if_good(X) = number_of_ancestors(X)  (TRUE)\n+-------------\n+\n+And this is true because at step 1) b) we remove the ancestors of the\n+\"good\" commits.\n+\n+If X is found to be \"bad\", then we know that its descendants are all\n+\"bad\", so we want to say that:\n+\n+-------------\n+information_if_bad(X) = number_of_descendants(X)  (WRONG)\n+-------------\n+\n+But this is wrong because at step 1) a) we keep only the ancestors of\n+the bad commit. So we get more information when a commit is marked as\n+\"bad\", because we also know that the ancestors of the previous \"bad\"\n+commit that are not ancestors of the new \"bad\" commit are not the\n+first bad commit. We don't know if they are good or bad, but we know\n+that they are not the first bad commit because they are not ancestor\n+of the new \"bad\" commit.\n+\n+So when a commit is marked as \"bad\" we know we can remove all the\n+commits in the graph except those that are ancestors of the new \"bad\"\n+commit. This means that:\n+\n+-------------\n+information_if_bad(X) = N - number_of_ancestors(X)  (TRUE)\n+-------------\n+\n+where N is the number of commits in the (cleaned up) graph.\n+\n+So in the end this means that to find the best bisection commits we\n+should maximize the function:\n+\n+-------------\n+f(X) = min(number_of_ancestors(X), N - number_of_ancestors(X))\n+-------------\n+\n+And this is nice because at step 2) we compute number_of_ancestors(X)\n+and so at step 3) we compute f(X).\n+\n+Let's take the following graph as an example:\n+\n+-------------\n+            G-H-I-J\n+           /       \\\n+A-B-C-D-E-F         O\n+           \\       /\n+            K-L-M-N\n+-------------\n+\n+If we compute the following non optimal function on it:\n+\n+-------------\n+g(X) = min(number_of_ancestors(X), number_of_descendants(X))\n+-------------\n+\n+we get:\n+\n+-------------\n+            4 3 2 1\n+            G-H-I-J\n+1 2 3 4 5 6/       \\0\n+A-B-C-D-E-F         O\n+           \\       /\n+            K-L-M-N\n+            4 3 2 1\n+-------------\n+\n+but with the algorithm used by git bisect we get:\n+\n+-------------\n+            7 7 6 5\n+            G-H-I-J\n+1 2 3 4 5 6/       \\0\n+A-B-C-D-E-F         O\n+           \\       /\n+            K-L-M-N\n+            7 7 6 5\n+-------------\n+\n+So we chose G, H, K or L as the best bisection point, which is better\n+than F. Because if for example L is bad, then we will know not only\n+that L, M and N are bad but also that G, H, I and J are not the first\n+bad commit (since we suppose that there is only one first bad commit\n+and it must be an ancestor of L).\n+\n+So the current algorithm seems to be the best possible given what we\n+initially supposed.\n+\n+Skip algorithm\n+~~~~~~~~~~~~~~\n+\n+When some commits have been skipped (using \"git bisect skip\"), then\n+the bisection algorithm is the same for step 1) to 3). But then we use\n+roughly the following steps:\n+\n+6) sort the commit by decreasing associated value\n+\n+7) if the first commit has not been skipped, we can return it and stop\n+here\n+\n+8) otherwise filter out all the skipped commits in the sorted list\n+\n+9) use a pseudo random number generator (PRNG) to generate a random\n+number between 0 and 1\n+\n+10) multiply this random number with its square root to bias it toward\n+0\n+\n+11) multiply the result by the number of commits in the filtered list\n+to get an index into this list\n+\n+12) return the commit at the computed index\n+\n+Skip algorithm discussed\n+~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+After step 7) (in the skip algorithm), we could check if the second\n+commit has been skipped and return it if it is not the case. And in\n+fact that was the algorithm we used from when \"git bisect skip\" was\n+developed in git version 1.5.4 (released on February 1st 2008) until\n+git version 1.6.4 (released July 29th 2009).\n+\n+But Ingo Molnar and H. Peter Anvin (another well known linux kernel\n+developer) both complained that sometimes the best bisection points\n+all happened to be in an area where all the commits are\n+untestable. And in this case the user was asked to test many\n+untestable commits, which could be very inefficient.\n+\n+Indeed untestable commits are often untestable because a breakage was\n+introduced at one time, and that breakage was fixed only after many\n+other commits were introduced.\n+\n+This breakage is of course most of the time unrelated to the breakage\n+we are trying to locate in the commit graph. But it prevents us to\n+know if the interesting \"bad behavior\" is present or not.\n+\n+So it is a fact that commits near an untestable commit have a high\n+probability of being untestable themselves. And the best bisection\n+commits are often found together too (due to the bisection algorithm).\n+\n+This is why it is a bad idea to just chose the next best unskipped\n+bisection commit when the first one has been skipped.\n+\n+We found that most commits on the graph may give quite a lot of\n+information when they are tested. And the commits that will not on\n+average give a lot of information are the one near the good and bad\n+commits.\n+\n+So using a PRNG with a bias to favor commits away from the good and\n+bad commits looked like a good choice.\n+\n+One obvious improvement to this algorithm would be to look for a\n+commit that has an associated value near the one of the best bisection\n+commit, and that is on another branch, before using the PRNG. Because\n+if such a commit exists, then it is not very likely to be untestable\n+too, so it will probably give more information than a nearly randomly\n+chosen one.\n+\n+Checking merge bases\n+~~~~~~~~~~~~~~~~~~~~\n+\n+There is another tweak in the bisection algorithm that has not been\n+described in the \"bisection algorithm\" above.\n+\n+We supposed in the previous examples that the \"good\" commits were\n+ancestors of the \"bad\" commit. But this is not a requirement of \"git\n+bisect\".\n+\n+Of course the \"bad\" commit cannot be an ancestor of a \"good\" commit,\n+because the ancestors of the good commits are supposed to be\n+\"good\". And all the \"good\" commits must be related to the bad commit.\n+They cannot be on a branch that has no link with the branch of the\n+\"bad\" commit. But it is possible for a good commit to be related to a\n+bad commit and yet not be neither one of its ancestor nor one of its\n+descendants.\n+\n+For example, there can be a \"main\" branch, and a \"dev\" branch that was\n+forked of the main branch at a commit named \"D\" like this:\n+\n+-------------\n+A-B-C-D-E-F-G  <--main\n+       \\\n+        H-I-J  <--dev\n+-------------\n+\n+The commit \"D\" is called a \"merge base\" for branch \"main\" and \"dev\"\n+because it's the best common ancestor for these branches for a merge.\n+\n+Now let's suppose that commit J is bad and commit G is good and that\n+we apply the bisection algorithm like it has been previously\n+described.\n+\n+As described in step 1) b) of the bisection algorithm, we remove all\n+the ancestors of the good commits because they are supposed to be good\n+too.\n+\n+So we would be left with only:\n+\n+-------------\n+H-I-J\n+-------------\n+\n+But what happens if the first bad commit is \"B\" and if it has been\n+fixed in the \"main\" branch by commit \"F\"?\n+\n+The result of such a bisection would be that we would find that H is\n+the first bad commit, when in fact it's B. So that would be wrong!\n+\n+And yes it's can happen in practice that people working on one branch\n+are not aware that people working on another branch fixed a bug! It\n+could also happen that F fixed more than one bug or that it is a\n+revert of some big development effort that was not ready to be\n+released.\n+\n+In fact development teams often maintain both a development branch and\n+a maintenance branch, and it would be quite easy for them if \"git\n+bisect\" just worked when they want to bisect a regression on the\n+development branch that is not on the maintenance branch. They should\n+be able to start bisecting using:\n+\n+-------------\n+$ git bisect start dev main\n+-------------\n+\n+To enable that additional nice feature, when a bisection is started\n+and when some good commits are not ancestors of the bad commit, we\n+first compute the merge bases between the bad and the good commits and\n+we chose these merge bases as the first commits that will be checked\n+out and tested.\n+\n+If it happens that one merge base is bad, then the bisection process\n+is stopped with a message like:\n+\n+-------------\n+The merge base BBBBBB is bad.\n+This means the bug has been fixed between BBBBBB and [GGGGGG,...].\n+-------------\n+\n+where BBBBBB is the sha1 hash of the bad merge base and [GGGGGG,...]\n+is a comma separated list of the sha1 of the good commits.\n+\n+If some of the merge bases are skipped, then the bisection process\n+continues, but the following message is printed for each skipped merge\n+base:\n+\n+-------------\n+Warning: the merge base between BBBBBB and [GGGGGG,...] must be skipped.\n+So we cannot be sure the first bad commit is between MMMMMM and BBBBBB.\n+We continue anyway.\n+-------------\n+\n+where BBBBBB is the sha1 hash of the bad commit, MMMMMM is the sha1\n+hash of the merge base that is skipped and [GGGGGG,...]  is a comma\n+separated list of the sha1 of the good commits.\n+\n+So if there is no bad merge base, the bisection process continues as\n+usual after this step.\n+\n+Best bisecting practices\n+------------------------\n+\n+Using test suites and git bisect together\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+If you both have a test suite and use git bisect, then it becomes less\n+important to check that all tests pass after each commit. Though of\n+course it is probably a good idea to have some checks to avoid\n+breaking too many things because it could make bisecting other bugs\n+more difficult.\n+\n+You can focus your efforts to check at a few points (for example rc\n+and beta releases) that all the T test cases pass for all the N\n+configurations. And when some tests don't pass you can use \"git\n+bisect\" (or better \"git bisect run\"). So you should perform roughly:\n+\n+-------------\n+c * N * T + b * M * log2(M) tests\n+-------------\n+\n+where c is the number of rounds of test (so a small constant) and b is\n+the ratio of bug per commit (hopefully a small constant too).\n+\n+So of course it's much better as it's O(N \\* T) vs O(N \\* T \\* M) if\n+you would test everything after each commit.\n+\n+This means that test suites are good to prevent some bugs from being\n+committed and they are also quite good to tell you that you have some\n+bugs. But they are not so good to tell you where some bugs have been\n+introduced. To tell you that efficiently, git bisect is needed.\n+\n+The other nice thing with test suites, is that when you have one, you\n+already know how to test for bad behavior. So you can use this\n+knowledge to create a new test case for \"git bisect\" when it appears\n+that there is a regression. So it will be easier to bisect the bug and\n+fix it. And then you can add the test case you just created to your\n+test suite.\n+\n+So if you know how to create test cases and how to bisect, you will be\n+subject to a virtuous circle:\n+\n+more tests => easier to create tests => easier to bisect => more tests\n+\n+So test suites and \"git bisect\" are complementary tools that are very\n+powerful and efficient when used together.\n+\n+Bisecting build failures\n+~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+You can very easily automatically bisect broken builds using something\n+like:\n+\n+-------------\n+$ git bisect start BAD GOOD\n+$ git bisect run make\n+-------------\n+\n+Passing sh -c \"some commands\" to \"git bisect run\"\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+For example:\n+\n+-------------\n+$ git bisect run sh -c \"make || exit 125; ./my_app | grep 'good output'\"\n+-------------\n+\n+On the other hand if you do this often, then it can be worth having\n+scripts to avoid too much typing.\n+\n+Finding performance regressions\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Here is an example script that comes slightly modified from a real\n+world script used by Junio Hamano <<4>>.\n+\n+This script can be passed to \"git bisect run\" to find the commit that\n+introduced a performance regression:\n+\n+-------------\n+#!/bin/sh\n+\n+# Build errors are not what I am interested in.\n+make my_app || exit 255\n+\n+# We are checking if it stops in a reasonable amount of time, so\n+# let it run in the background...\n+\n+./my_app >log 2>&1 &\n+\n+# ... and grab its process ID.\n+pid=$!\n+\n+# ... and then wait for sufficiently long.\n+sleep $NORMAL_TIME\n+\n+# ... and then see if the process is still there.\n+if kill -0 $pid\n+then\n+\t# It is still running -- that is bad.\n+\tkill $pid; sleep 1; kill $pid;\n+\texit 1\n+else\n+\t# It has already finished (the $pid process was no more),\n+        # and we are happy.\n+\texit 0\n+fi\n+-------------\n+\n+Following general best practices\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+It is obviously a good idea not to have commits with changes that\n+knowingly break things, even if some other commits later fix the\n+breakage.\n+\n+It is also a good idea when using any VCS to have only one small\n+logical change in each commit.\n+\n+The smaller the changes in your commit, the most effective \"git\n+bisect\" will be. And you will probably need \"git bisect\" less in the\n+first place, as small changes are easier to review even if they are\n+only reviewed by the commiter.\n+\n+Another good idea is to have good commit messages. They can be very\n+helpful to understand why some changes were made.\n+\n+These general best practices are very helpful if you bisect often.\n+\n+Avoiding bug prone merges\n+~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+First merges by themselves can introduce some regressions even when\n+the merge needs no source code conflict resolution. This is because a\n+semantic change can happen in one branch while the other branch is not\n+aware of it.\n+\n+For example one branch can change the semantic of a function while the\n+other branch add more calls to the same function.\n+\n+This is made much worse if many files have to be fixed to resolve\n+conflicts. That's why such merges are called \"evil merges\". They can\n+make regressions very difficult to track down. It can even be\n+misleading to know the first bad commit if it happens to be such a\n+merge, because people might think that the bug comes from bad conflict\n+resolution when it comes from a semantic change in one branch.\n+\n+Anyway \"git rebase\" can be used to linearize history. This can be used\n+either to avoid merging in the first place. Or it can be used to\n+bisect on a linear history instead of the non linear one, as this\n+should give more information in case of a semantic change in one\n+branch.\n+\n+Merges can be also made simpler by using smaller branches or by using\n+many topic branches instead of only long version related branches.\n+\n+And testing can be done more often in special integration branches\n+like linux-next for the linux kernel.\n+\n+Adapting your work-flow\n+~~~~~~~~~~~~~~~~~~~~~~~\n+\n+A special work-flow to process regressions can give great results.\n+\n+Here is an example of a work-flow used by Andreas Ericsson:\n+\n+* write, in the test suite, a test script that exposes the regression\n+* use \"git bisect run\" to find the commit that introduced it\n+* fix the bug that is often made obvious by the previous step\n+* commit both the fix and the test script (and if needed more tests)\n+\n+And here is what Andreas said about this work-flow <<5>>:\n+\n+_____________\n+To give some hard figures, we used to have an average report-to-fix\n+cycle of 142.6 hours (according to our somewhat weird bug-tracker\n+which just measures wall-clock time). Since we moved to git, we've\n+lowered that to 16.2 hours. Primarily because we can stay on top of\n+the bug fixing now, and because everyone's jockeying to get to fix\n+bugs (we're quite proud of how lazy we are to let git find the bugs\n+for us). Each new release results in ~40% fewer bugs (almost certainly\n+due to how we now feel about writing tests).\n+_____________\n+\n+Clearly this work-flow uses the virtuous circle between test suites\n+and \"git bisect\". In fact it makes it the standard procedure to deal\n+with regression.\n+\n+In other messages Andreas says that they also use the \"best practices\"\n+described above: small logical commits, topic branches, no evil\n+merge,... These practices all improve the bisectability of the commit\n+graph, by making it easier and more useful to bisect.\n+\n+So a good work-flow should be designed around the above points. That\n+is making bisecting easier, more useful and standard.\n+\n+Involving QA people and if possible end users\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+One nice about \"git bisect\" is that it is not only a developer\n+tool. It can effectively be used by QA people or even end users (if\n+they have access to the source code or if they can get access to all\n+the builds).\n+\n+There was a discussion at one point on the linux kernel mailing list\n+of whether it was ok to always ask end user to bisect, and very good\n+points were made to support the point of view that it is ok.\n+\n+For example David Miller wrote <<6>>:\n+\n+_____________\n+What people don't get is that this is a situation where the \"end node\n+principle\" applies. When you have limited resources (here: developers)\n+you don't push the bulk of the burden upon them. Instead you push\n+things out to the resource you have a lot of, the end nodes (here:\n+users), so that the situation actually scales.\n+_____________\n+\n+This means that it is often \"cheaper\" if QA people or end users can do\n+it.\n+\n+What is interesting too is that end users that are reporting bugs (or\n+QA people that reproduced a bug) have access to the environment where\n+the bug happens. So they can often more easily reproduce a\n+regression. And if they can bisect, then more information will be\n+extracted from the environment where the bug happens, which means that\n+it will be easier to understand and then fix the bug.\n+\n+For open source projects it can be a good way to get more useful\n+contributions from end users, and to introduce them to QA and\n+development activities.\n+\n+Using complex scripts\n+~~~~~~~~~~~~~~~~~~~~~\n+\n+In some cases like for kernel development it can be worth developing\n+complex scripts to be able to fully automate bisecting.\n+\n+Here is what Ingo Molnar says about that <<7>>:\n+\n+_____________\n+i have a fully automated bootup-hang bisection script. It is based on\n+\"git-bisect run\". I run the script, it builds and boots kernels fully\n+automatically, and when the bootup fails (the script notices that via\n+the serial log, which it continuously watches - or via a timeout, if\n+the system does not come up within 10 minutes it's a \"bad\" kernel),\n+the script raises my attention via a beep and i power cycle the test\n+box. (yeah, i should make use of a managed power outlet to 100%\n+automate it)\n+_____________\n+\n+Combining test suites, git bisect and other systems together\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+We have seen that test suites an git bisect are very powerful when\n+used together. It can be even more powerful if you can combine them\n+with other systems.\n+\n+For example some test suites could be run automatically at night with\n+some unusual (or even random) configurations. And if a regression is\n+found by a test suite, then \"git bisect\" can be automatically\n+launched, and its result can be emailed to the author of the first bad\n+commit found by \"git bisect\", and perhaps other people too. And a new\n+entry in the bug tracking system could be automatically created too.\n+\n+\n+The future of bisecting\n+-----------------------\n+\n+\"git replace\"\n+~~~~~~~~~~~~~\n+\n+We saw earlier that \"git bisect skip\" is now using a PRNG to try to\n+avoid areas in the commit graph where commits are untestable. The\n+problem is that sometimes the first bad commit will be in an\n+untestable area.\n+\n+To simplify the discussion we will suppose that the untestable area is\n+a simple string of commits and that it was created by a breakage\n+introduced by one commit (let's call it BBC for bisect breaking\n+commit) and later fixed by another one (let's call it BFC for bisect\n+fixing commit).\n+\n+For example:\n+\n+-------------\n+...-Y-BBC-X1-X2-X3-X4-X5-X6-BFC-Z-...\n+-------------\n+\n+where we know that Y is good and BFC is bad, and where BBC and X1 to\n+X6 are untestable.\n+\n+In this case if you are bisecting manually, what you can do is create\n+a special branch that starts just before the BBC. The first commit in\n+this branch should be the BBC with the BFC squashed into it. And the\n+other commits in the branch should be the commits between BBC and BFC\n+rebased on the first commit of the branch and then the commit after\n+BFC also rebased on.\n+\n+For example:\n+\n+-------------\n+      (BBC+BFC)-X1'-X2'-X3'-X4'-X5'-X6'-Z'\n+     /\n+...-Y-BBC-X1-X2-X3-X4-X5-X6-BFC-Z-...\n+-------------\n+\n+where commits quoted with ' have been rebased.\n+\n+You can easily create such a branch with Git using interactive rebase.\n+\n+For example using:\n+\n+-------------\n+$ git rebase -i Y Z\n+-------------\n+\n+and then moving BFC after BBC and squashing it.\n+\n+After that you can start bisecting as usual in the new branch and you\n+should eventually find the first bad commit.\n+\n+For example:\n+\n+-------------\n+$ git bisect start Z' Y\n+-------------\n+\n+If you are using \"git bisect run\", you can use the same manual fix up\n+as above, and then start another \"git bisect run\" in the special\n+branch. Or as the \"git bisect\" man page says, the script passed to\n+\"git bisect run\" can apply a patch before it compiles and test the\n+software <<8>>. The patch should turn a current untestable commits\n+into a testable one. So the testing will result in \"good\" or \"bad\" and\n+\"git bisect\" will be able to find the first bad commit. And the script\n+should not forget to remove the patch once the testing is done before\n+exiting from the script.\n+\n+(Note that instead of a patch you can use \"git cherry-pick BFC\" to\n+apply the fix, and in this case you should use \"git reset --hard\n+HEAD^\" to revert the cherry-pick after testing and before returning\n+from the script.)\n+\n+But the above ways to work around untestable areas are a little bit\n+clunky. Using special branches is nice because these branches can be\n+shared by developers like usual branches, but the risk is that people\n+will get many such branches. And it disrupts the normal \"git bisect\"\n+work-flow. So, if you want to use \"git bisect run\" completely\n+automatically, you have to add special code in your script to restart\n+bisection in the special branches.\n+\n+Anyway one can notice in the above special branch example that the Z'\n+and Z commits should point to the same source code state (the same\n+\"tree\" in git parlance). That's because Z' result from applying the\n+same changes as Z just in a slightly different order.\n+\n+So if we could just \"replace\" Z by Z' when we bisect, then we would\n+not need to add anything to a script. It would just work for anyone in\n+the project sharing the special branches and the replacements.\n+\n+With the example above that would give:\n+\n+-------------\n+      (BBC+BFC)-X1'-X2'-X3'-X4'-X5'-X6'-Z'-...\n+     /\n+...-Y-BBC-X1-X2-X3-X4-X5-X6-BFC-Z\n+-------------\n+\n+That's why the \"git replace\" command was created. Technically it\n+stores replacements \"refs\" in the \"refs/replace/\" hierarchy. These\n+\"refs\" are like branches (that are stored in \"refs/heads/\") or tags\n+(that are stored in \"refs/tags\"), and that means that they can\n+automatically be shared like branches or tags among developers.\n+\n+\"git replace\" is a very powerful mechanism. It can be used to fix\n+commits in already released history, for example to change the commit\n+message or the author. And it can also be used instead of git \"grafts\"\n+to link a repository with another old repository.\n+\n+In fact it's this last feature that \"sold\" it to the git community, so\n+it is now in the \"master\" branch of git's git repository and it should\n+be released in git 1.6.5 in October or November 2009.\n+\n+One problem with \"git replace\" is that currently it stores all the\n+replacements refs in \"refs/replace/\", but it would be perhaps better\n+if the replacement refs that are useful only for bisecting would be in\n+\"refs/replace/bisect/\". This way the replacement refs could be used\n+only for bisecting, while other refs directly in \"refs/replace/\" would\n+be used nearly all the time.\n+\n+Bisecting sporadic bugs\n+~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Another possible improvement to \"git bisect\" would be to optionally\n+add some redundancy to the tests performed so that it would be more\n+reliable when tracking sporadic bugs.\n+\n+This has been requested by some kernel developers because some bugs\n+called sporadic bugs do not appear in all the kernel builds because\n+they are very dependent on the compiler output.\n+\n+The idea is that every 3 test for example, \"git bisect\" could ask the\n+user to test a commit that has already been found to be \"good\" or\n+\"bad\" (because one of its descendants or one of its ancestors has been\n+found to be \"good\" or \"bad\" respectively). If it happens that a commit\n+has been previously incorrectly classified then the bisection can be\n+aborted early, hopefully before too many mistakes have been made. Then\n+the user will have to look at what happened and then restart the\n+bisection using a fixed bisect log.\n+\n+There is already a project called BBChop created by Ealdwulf Wuffinga\n+on Github that does something like that using Bayesian Search Theory\n+<<9>>:\n+\n+_____________\n+BBChop is like 'git bisect' (or equivalent), but works when your bug\n+is intermittent. That is, it works in the presence of false negatives\n+(when a version happens to work this time even though it contains the\n+bug). It assumes that there are no false positives (in principle, the\n+same approach would work, but adding it may be non-trivial).\n+_____________\n+\n+But BBChop is independent of any VCS and it would be easier for Git\n+users to have something integrated in Git.\n+\n+Conclusion\n+----------\n+\n+We have seen that regressions are an important problem, and that \"git\n+bisect\" has nice features that complement very well practices and\n+other tools, especially test suites, that are generally used to fight\n+regressions. But it might be needed to change some work-flows and\n+(bad) habits to get the most out of it.\n+\n+Some improvements to the algorithms inside \"git bisect\" are possible\n+and some new features could help in some cases, but overall \"git\n+bisect\" works already very well, is used a lot, and is already very\n+useful. To back up that last claim, let's give the final word to Ingo\n+Molnar when he was asked by the author how much time does he think\n+\"git bisect\" saves him when he uses it:\n+\n+_____________\n+a _lot_.\n+\n+About ten years ago did i do my first 'bisection' of a Linux patch\n+queue. That was prior the Git (and even prior the BitKeeper) days. I\n+literally days spent sorting out patches, creating what in essence\n+were standalone commits that i guessed to be related to that bug.\n+\n+It was a tool of absolute last resort. I'd rather spend days looking\n+at printk output than do a manual 'patch bisection'.\n+\n+With Git bisect it's a breeze: in the best case i can get a ~15 step\n+kernel bisection done in 20-30 minutes, in an automated way. Even with\n+manual help or when bisecting multiple, overlapping bugs, it's rarely\n+more than an hour.\n+\n+In fact it's invaluable because there are bugs i would never even\n+_try_ to debug if it wasn't for git bisect. In the past there were bug\n+patterns that were immediately hopeless for me to debug - at best i\n+could send the crash/bug signature to lkml and hope that someone else\n+can think of something.\n+\n+And even if a bisection fails today it tells us something valuable\n+about the bug: that it's non-deterministic - timing or kernel image\n+layout dependent.\n+\n+So git bisect is unconditional goodness - and feel free to quote that \n+;-)\n+_____________\n+\n+Acknowledgements\n+----------------\n+\n+Many thanks to Junio Hamano for his help in reviewing this paper, for\n+reviewing the patches I sent to the git mailing list, for discussing\n+some ideas and helping me improve them, for improving \"git bisect\" a\n+lot and for his awesome work in maintaining and developing Git.\n+\n+Many thanks to Ingo Molnar for giving me very useful information that\n+appears in this paper, for commenting on this paper, for his\n+suggestions to improve \"git bisect\" and for evangelizing \"git bisect\"\n+on the linux kernel mailing lists.\n+\n+Many thanks to Linus Torvalds for inventing, developing and\n+evangelizing \"git bisect\", Git and Linux.\n+\n+Many thanks to the many other great people who helped one way or\n+another when I worked on git, especially to Andreas Ericsson, Johannes\n+Schindelin, H. Peter Anvin, Daniel Barkalow, Bill Lear, John Hawley,\n+Shawn O. Pierce, Jeff King, Sam Vilain, Jon Seymour.\n+\n+Many thanks to the Linux-Kongress program committee for choosing the\n+author to given a talk and for publishing this paper.\n+\n+References\n+----------\n+\n+- [[[1]]] http://www.nist.gov/public_affairs/releases/n02-10.htm['Software Errors Cost U.S. Economy $59.5 Billion Annually'. Nist News Release.]\n+- [[[2]]] http://java.sun.com/docs/codeconv/html/CodeConventions.doc.html#16712['Code Conventions for the Java Programming Language'. Sun Microsystems.]\n+- [[[3]]] http://en.wikipedia.org/wiki/Software_maintenance['Software maintenance'. Wikipedia.]\n+- [[[4]]] http://article.gmane.org/gmane.comp.version-control.git/45195/[Junio C Hamano. 'Automated bisect success story'. Gmane.]\n+- [[[5]]] http://lwn.net/Articles/317154/[Christian Couder. 'Fully automated bisecting with \"git bisect run\"'. LWN.net.]\n+- [[[6]]] http://lwn.net/Articles/277872/[Jonathan Corbet. 'Bisection divides users and developers'. LWN.net.]\n+- [[[7]]] http://article.gmane.org/gmane.linux.scsi/36652/[Ingo Molnar. 'Re: BUG 2.6.23-rc3 can't see sd partitions on Alpha'. Gmane.]\n+- [[[8]]] http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html[Junio C Hamano and the git-list. 'git-bisect(1) Manual Page'. Linux Kernel Archives.]\n+- [[[9]]] http://github.com/Ealdwulf/bbchop[Ealdwulf. 'bbchop'. GitHub.]\n+\ndiff --git a/Documentation/git-bisect.txt b/Documentation/git-bisect.txt\nindex d2ffae0..c39d957 100644\n--- a/Documentation/git-bisect.txt\n+++ b/Documentation/git-bisect.txt\n@@ -330,6 +330,11 @@ Documentation\n -------------\n Documentation by Junio C Hamano and the git-list <git@vger.kernel.org>.\n \n+SEE ALSO\n+--------\n+link:git-bisect-lk2009.html[Fighting regressions with git bisect],\n+linkgit:git-blame[1].\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\n-- \n1.6.5.1.gaf97d\n"}]}