{"thread":{"id":"17580","subject":"Article about \"git bisect run\" on LWN","startedAt":"2009-02-05T06:47:49Z","lastAt":"2009-02-10T06:12:31Z","messageCount":18,"participants":["Christian Couder","Bill Lear","Ingo Molnar","Jonathan Corbet","david@lang.hm","David Symonds","Junio C Hamano","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"103276","messageId":"200902050747.50100.chriscool@tuxfamily.org","threadId":"17580","inReplyTo":null,"subject":"Article about \"git bisect run\" on LWN","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2009-02-05T06:47:49Z","receivedAt":"2009-02-05T06:47:49Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nFor information, an article from me, 'Fully automated bisecting with \"git \nbisect run\"' has been published in today's edition of LWN on the \ndevelopment page:\n\nhttp://lwn.net/Articles/317154/\n\nThank you guys,\nChristian.\n"},{"id":"103333","messageId":"18826.60142.732388.27262@lisa.zopyra.com","threadId":"17580","inReplyTo":"200902050747.50100.chriscool@tuxfamily.org","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Bill Lear","fromEmail":"rael@zopyra.com","sentAt":"2009-02-05T13:34:38Z","receivedAt":"2009-02-05T13:34:38Z","isPatch":false,"sender":{"key":"rael@zopyra.com","avatar":"https://gravatar.com/avatar/c4f2d2790ca3828d3b4e7dfebabf61d2fe94fd82fa49cdac2a5295dd2d46a874?d=mp&s=160"},"body":"On Thursday, February 5, 2009 at 07:47:49 (+0100) Christian Couder writes:\n>Hi,\n>\n>For information, an article from me, 'Fully automated bisecting with \"git \n>bisect run\"' has been published in today's edition of LWN on the \n>development page:\n>\n>http://lwn.net/Articles/317154/\n\nWow, very nice Christian. Thank you for including me in the article\nand for sending me this notice --- very thoughtful of you.\n\n\nBill\n"},{"id":"103336","messageId":"20090205141336.GA28443@elte.hu","threadId":"17580","inReplyTo":"200902050747.50100.chriscool@tuxfamily.org","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Ingo Molnar","fromEmail":"mingo@elte.hu","sentAt":"2009-02-05T14:13:36Z","receivedAt":"2009-02-05T14:13:36Z","isPatch":false,"sender":{"key":"mingo@elte.hu","avatar":null},"body":"\n* Christian Couder wrote:\n\n> Hi,\n> \n> For information, an article from me, 'Fully automated bisecting with \"git \n> bisect run\"' has been published in today's edition of LWN on the \n> development page:\n> \n> http://lwn.net/Articles/317154/\n\nNice article!\n\nIn terms of possible future enhancements of git bisect, here's a couple of \nrandom ideas that would help my auto-bisection efforts:\n\n - Feature: support \"Bisection Redundancy\"\n\n This feature helps developers realize if a bug is sporadic. This happens \n quite often in the kernel space: a bug looks deterministic, but down the \n line it becomes sporadic. Sometimes a boot crash only occurs with a 75% \n probability - and if one is unlucky it can cause a _lot_ of wasted \n bisection time. The wrong commit gets blamed and the wrong set of \n developers start scratching their heads. It's a reoccuring theme on lkml.\n\n What git could do here is to allow testers to inject a bit of extra \n \"redundancy\" automatically, and use the redundant test-points to detect \n conflicts in good/bad constraints.\n\n It would work like this:\n\n git bisect start --redundancy=33%\n\n It would mean that for every third bisection points, Git would\n _not_ chose the ideal (estimated) 'middle point' from the set of \"unknown \n quality\" changes that are still outstanding - but would intentionally \n \"weer outside\" and select one commit from the _known_ set of commits.\n\n If such a redundant re-test of the known-good or known-bad set yields a \n nonsensical result then Git aborts the bisection with a \"logic\n inconsistency detected\" kind of message - and people could at this point\n realize the non-determinism of the test.\n\n ( Git can do this when a \"redundant\" test point is marked as 'bad' - \n despite an earlier bisection already categorizing that test point as \n 'good' - or if it's the other way around. Git will only continue with \n the bisection if the test point has the expected quality. )\n\n This essentially means an automated re-test - but it's much better than \n just a repeated bisection - i've often met non-deterministic bugs that \n yield the _exact same_ nonsensical commit even on repeat bisections. That \n happens when a timing bug depends on the exact kernel layout, or a \n miscompilation or linker bug depends on the exact kernel layout, etc.\n\n It's also faster than a re-done bisection: 33% more testpoints is better \n than twice as many test-points. Also, auto-bisection can deal with \n redundancy just fine - it does not really matter whether i have to wait \n 20 or 30 minutes for a test result since there's no manual intervention \n needed - but it _very_ much matters whether i can trust the validity of \n the bisection result.\n\n- Feature: better \"git bisect next\" support.\n\n Sometimes a commit wont build. In that case we have \"git bisect next\", but \n last i checked that only jumps a single commit - and build breakages \n often have a large scope - full trees that got merged upstream, etc. Most \n of the time those build breakages are uninteresting and the build-broken \n window does not contain the bad commit.\n\n So it would be nice to have a \"git bisect next --left=20%\" type of \n feature. This would jump 20% commits to the \"left\" from the bisection \n point, towards the 'known bad' set of commits, but still within the \n bisection window.\n\n Similarly, \"git bisect next --right=20%\" would jump towards the known-good \n edge of the bisection window (but still within the bisection window).\n\n Currently when i hit a build error during auto-bisection, it aborts and i \n have to intervene manually. But with a bigger jump distance i could use\n git-bisect-next reliably in scripts too.\n\n Likewise, users too hit build breakages often, and find it hard to get out \n of the window of breakage. With the high-order tree structure of the \n kernel repository that is rather non-intuitive to do as well, and often \n people make mistakes and test the wrong commit.\n\n- Feature: detect \"redundant\" and \"inconsistent\" test points\n\n This is a variation of the redunant testing theme, but from a different \n angle: often newbies when they bisect the kernel weer outside of the \n bisection window without realizing it. It would be nice if Git printed a \n friendly notifier that:\n\n git bisect good 12341234\n info: bisection point 12341234 was already in the 'good' range\n\n Or, if the redunant test point is conflicting, print:\n\n git bisect good 12341234\n fatal: bisection point 12341234 was already in the 'bad' range!\n\n And give an error return as well, so that scripts can abort.\n\n Currently Git seems to be very forgiving and accepts all bisection points \n that we feed it, without checking them for consistency. (this might have \n changed in current development versions, i dont know.)\n\n- User friendliness: give an estimation about how many steps are remaining\n\n Right now git prints this when a bisection session begins:\n\n aldebaran:~/tip> git bisect start\n aldebaran:~/tip> git bisect bad linus\n aldebaran:~/tip> git bisect good v2.6.28\n Bisecting: 5449 revisions left to test after this\n [e0b685d39a0404e7f87fb7b7808c3b37a115fe11] Updated contact info for CREDITS file\n\n It would be nice if Git estimated the expected number of bisection points. \n Something like this would be helpful:\n\n aldebaran:~/tip> git bisect good v2.6.28\n Bisecting: 5449 revisions left to test after this\n About ~16 test steps left [approximated]\n [e0b685d39a0404e7f87fb7b7808c3b37a115fe11] Updated contact info for CREDITS file\n\n The real number of test points might be higher than this, if the tree \n layout is unlucky, or it might be less than this if the user manually \n narrows the bisection window to a suspected set of commits - but that's \n OK - most kernel testers use the default variant and the message is clear \n enough that it's only an estimation.\n\n\tIngo\n"},{"id":"103347","messageId":"20090205092319.1a3d5d8b@bike.lwn.net","threadId":"17580","inReplyTo":"200902050747.50100.chriscool@tuxfamily.org","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Jonathan Corbet","fromEmail":"corbet@lwn.net","sentAt":"2009-02-05T16:23:19Z","receivedAt":"2009-02-05T16:23:19Z","isPatch":false,"sender":{"key":"corbet@lwn.net","avatar":null},"body":"On Thu, 5 Feb 2009 07:47:49 +0100\nChristian Couder wrote:\n\n> For information, an article from me, 'Fully automated bisecting with\n> \"git bisect run\"' has been published in today's edition of LWN on the \n> development page:\n> \n> http://lwn.net/Articles/317154/\n\nFor those of you lacking an LWN subscription, please follow this link\ninstead:\n\n\thttp://lwn.net/SubscriberLink/317154/c49a5d1a14050820/\n\nThanks to Christian for doing this article for us!\n\njon\n"},{"id":"103389","messageId":"200902052154.41525.chriscool@tuxfamily.org","threadId":"17580","inReplyTo":"20090205092319.1a3d5d8b@bike.lwn.net","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2009-02-05T20:54:41Z","receivedAt":"2009-02-05T20:54:41Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Le jeudi 5 février 2009, Jonathan Corbet a écrit :\n> On Thu, 5 Feb 2009 07:47:49 +0100\n>\n> For those of you lacking an LWN subscription, please follow this link\n> instead:\n>\n> \thttp://lwn.net/SubscriberLink/317154/c49a5d1a14050820/\n>\n> Thanks to Christian for doing this article for us!\n\nThank you Jon for publishing this article and for the subscriber link.\nAnd thank you Jake for your help in writing this article.\n\nBest regards,\nChristian.\n"},{"id":"103417","messageId":"20090206014655.GA26807@elte.hu","threadId":"17580","inReplyTo":"alpine.DEB.1.10.0902051838180.5340@asgard.lang.hm","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Ingo Molnar","fromEmail":"mingo@elte.hu","sentAt":"2009-02-06T01:46:55Z","receivedAt":"2009-02-06T01:46:55Z","isPatch":false,"sender":{"key":"mingo@elte.hu","avatar":null},"body":"\n* david@lang.hm wrote:\n\n> On Thu, 5 Feb 2009, Ingo Molnar wrote:\n>\n>> * Christian Couder wrote:\n>>\n>>> Hi,\n>>>\n>>> For information, an article from me, 'Fully automated bisecting with \"git\n>>> bisect run\"' has been published in today's edition of LWN on the\n>>> development page:\n>>>\n>>> http://lwn.net/Articles/317154/\n>>\n>> Nice article!\n>>\n>> In terms of possible future enhancements of git bisect, here's a couple of\n>> random ideas that would help my auto-bisection efforts:\n>>\n>> - Feature: support \"Bisection Redundancy\"\n>>\n>> This feature helps developers realize if a bug is sporadic. This happens\n>> quite often in the kernel space: a bug looks deterministic, but down the\n>> line it becomes sporadic. Sometimes a boot crash only occurs with a 75%\n>> probability - and if one is unlucky it can cause a _lot_ of wasted\n>> bisection time. The wrong commit gets blamed and the wrong set of\n>> developers start scratching their heads. It's a reoccuring theme on lkml.\n>>\n>> What git could do here is to allow testers to inject a bit of extra\n>> \"redundancy\" automatically, and use the redundant test-points to detect\n>> conflicts in good/bad constraints.\n>>\n>> It would work like this:\n>>\n>> git bisect start --redundancy=33%\n>>\n>> It would mean that for every third bisection points, Git would\n>> _not_ chose the ideal (estimated) 'middle point' from the set of \"unknown\n>> quality\" changes that are still outstanding - but would intentionally\n>> \"weer outside\" and select one commit from the _known_ set of commits.\n>>\n>> If such a redundant re-test of the known-good or known-bad set yields a\n>> nonsensical result then Git aborts the bisection with a \"logic\n>> inconsistency detected\" kind of message - and people could at this point\n>> realize the non-determinism of the test.\n>>\n>> ( Git can do this when a \"redundant\" test point is marked as 'bad' -\n>> despite an earlier bisection already categorizing that test point as\n>> 'good' - or if it's the other way around. Git will only continue with\n>> the bisection if the test point has the expected quality. )\n>>\n>> This essentially means an automated re-test - but it's much better than\n>> just a repeated bisection - i've often met non-deterministic bugs that\n>> yield the _exact same_ nonsensical commit even on repeat bisections. That\n>> happens when a timing bug depends on the exact kernel layout, or a\n>> miscompilation or linker bug depends on the exact kernel layout, etc.\n>>\n>> It's also faster than a re-done bisection: 33% more testpoints is better\n>> than twice as many test-points. Also, auto-bisection can deal with\n>> redundancy just fine - it does not really matter whether i have to wait\n>> 20 or 30 minutes for a test result since there's no manual intervention\n>> needed - but it _very_ much matters whether i can trust the validity of\n>> the bisection result.\n>\n> when you gave this the title of redundnancy and described the problem I \n> assumed that you would then propose running the test multiple times (so \n> \"git bisect run X --redundancy 5\" would run each test 5 times, it would \n> pass IFF it passed the test all 5 times. that would seem to be a better \n> match for the name, as well as being a better test\n\nYeah, but using 100%, 200%, 300%, etc. redundancy is a bit wasteful and not \ngranular enough for my purposes.\n\nHere's the math:\n\nA typical kernel bisection takes 15 test steps. 30% of redundancy means that \nit takes only 30% longer, but for that we get +5 tests. Five extra test \npoints are usually enough to establish whether a test method shows sporadic \ntendencies or not, with an ~90% confidence factor.\n\nRepeating the test 5 times would bring a 15-steps kernel bisection from 30 \nminutes [it's about 60 seconds to build a kernel, 60 seconds to boot it] to \nabout 2.5 hours - that's very long. The confidence factor only goes from \n~90% to 99% - that extra 9% is not worth the cost.\n\nThe idea would be to insert 30% redunancy into my bisections automatically - \nso that i could trust _all_ bisections more - not just the ones i suspect to \nbe non-deterministic. Hence the suggestion to enable lower levels of \nredundancy like 30%. (but even 10% or 20% might be enough to weed out the \nmost obvious cases)\n\n\tIngo\n"},{"id":"103418","messageId":"20090206015215.GA6261@elte.hu","threadId":"17580","inReplyTo":"20090206014655.GA26807@elte.hu","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Ingo Molnar","fromEmail":"mingo@elte.hu","sentAt":"2009-02-06T01:52:15Z","receivedAt":"2009-02-06T01:52:15Z","isPatch":false,"sender":{"key":"mingo@elte.hu","avatar":null},"body":"\n* Ingo Molnar wrote:\n\n> The idea would be to insert 30% redunancy into my bisections automatically \n> - so that i could trust _all_ bisections more - not just the ones i \n> suspect to be non-deterministic. Hence the suggestion to enable lower \n> levels of redundancy like 30%. (but even 10% or 20% might be enough to \n> weed out the most obvious cases)\n\nthe other advantage of redundancy that i forgot to mention:\n\n- Sometimes the non-determinism is inserted by a _human_. It happened not\n once that i accidentally mis-judged a testpoint, and the bisection ran\n afoul. Only 4-5 steps later do i suspect that something is wrong: that i \n get an unlikely series of good,good,good,good,good or bad,bad,bad,bad,bad \n testpoint qualities.\n\nSo for manual bisection, redundancy can be a big time-saver. If i mess up a \nbisection point then say 50% redundancy can still point out my stupidity \nwith a high likelyhood.\n\nIn fact if Git sees an unlikely series of same-quality bisection points, it \ncould artificially insert a test to around the last-different test point, to \ntest the theory of a messed up bisection.\n\n\tIngo\n"},{"id":"103415","messageId":"alpine.DEB.1.10.0902051838180.5340@asgard.lang.hm","threadId":"17580","inReplyTo":"20090205141336.GA28443@elte.hu","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"","fromEmail":"david@lang.hm","sentAt":"2009-02-06T02:42:16Z","receivedAt":"2009-02-06T02:42:16Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Thu, 5 Feb 2009, Ingo Molnar wrote:\n\n> * Christian Couder wrote:\n>\n>> Hi,\n>>\n>> For information, an article from me, 'Fully automated bisecting with \"git\n>> bisect run\"' has been published in today's edition of LWN on the\n>> development page:\n>>\n>> http://lwn.net/Articles/317154/\n>\n> Nice article!\n>\n> In terms of possible future enhancements of git bisect, here's a couple of\n> random ideas that would help my auto-bisection efforts:\n>\n> - Feature: support \"Bisection Redundancy\"\n>\n> This feature helps developers realize if a bug is sporadic. This happens\n> quite often in the kernel space: a bug looks deterministic, but down the\n> line it becomes sporadic. Sometimes a boot crash only occurs with a 75%\n> probability - and if one is unlucky it can cause a _lot_ of wasted\n> bisection time. The wrong commit gets blamed and the wrong set of\n> developers start scratching their heads. It's a reoccuring theme on lkml.\n>\n> What git could do here is to allow testers to inject a bit of extra\n> \"redundancy\" automatically, and use the redundant test-points to detect\n> conflicts in good/bad constraints.\n>\n> It would work like this:\n>\n> git bisect start --redundancy=33%\n>\n> It would mean that for every third bisection points, Git would\n> _not_ chose the ideal (estimated) 'middle point' from the set of \"unknown\n> quality\" changes that are still outstanding - but would intentionally\n> \"weer outside\" and select one commit from the _known_ set of commits.\n>\n> If such a redundant re-test of the known-good or known-bad set yields a\n> nonsensical result then Git aborts the bisection with a \"logic\n> inconsistency detected\" kind of message - and people could at this point\n> realize the non-determinism of the test.\n>\n> ( Git can do this when a \"redundant\" test point is marked as 'bad' -\n> despite an earlier bisection already categorizing that test point as\n> 'good' - or if it's the other way around. Git will only continue with\n> the bisection if the test point has the expected quality. )\n>\n> This essentially means an automated re-test - but it's much better than\n> just a repeated bisection - i've often met non-deterministic bugs that\n> yield the _exact same_ nonsensical commit even on repeat bisections. That\n> happens when a timing bug depends on the exact kernel layout, or a\n> miscompilation or linker bug depends on the exact kernel layout, etc.\n>\n> It's also faster than a re-done bisection: 33% more testpoints is better\n> than twice as many test-points. Also, auto-bisection can deal with\n> redundancy just fine - it does not really matter whether i have to wait\n> 20 or 30 minutes for a test result since there's no manual intervention\n> needed - but it _very_ much matters whether i can trust the validity of\n> the bisection result.\n\nwhen you gave this the title of redundnancy and described the problem I \nassumed that you would then propose running the test multiple times (so \n\"git bisect run X --redundancy 5\" would run each test 5 times, it would \npass IFF it passed the test all 5 times. that would seem to be a better \nmatch for the name, as well as being a better test\n\nDavid Lang\n"},{"id":"103416","messageId":"alpine.DEB.1.10.0902051842580.5340@asgard.lang.hm","threadId":"17580","inReplyTo":"20090205092319.1a3d5d8b@bike.lwn.net","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"","fromEmail":"david@lang.hm","sentAt":"2009-02-06T02:49:56Z","receivedAt":"2009-02-06T02:49:56Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Thu, 5 Feb 2009, Jonathan Corbet wrote:\n\n> On Thu, 5 Feb 2009 07:47:49 +0100\n> Christian Couder wrote:\n>\n>> For information, an article from me, 'Fully automated bisecting with\n>> \"git bisect run\"' has been published in today's edition of LWN on the\n>> development page:\n>>\n>> http://lwn.net/Articles/317154/\n>\n> For those of you lacking an LWN subscription, please follow this link\n> instead:\n>\n> \thttp://lwn.net/SubscriberLink/317154/c49a5d1a14050820/\n>\n> Thanks to Christian for doing this article for us!\n\nand since Jon was too professional to put in a plug himself, I'll do it \nfor him ;-)\n\nif you aren't a subscriber, consider subscribing (the 'starving-hacker' \nlevel is only $2.50USD/month)\n\nDavid Lang\n"},{"id":"103427","messageId":"200902060623.16046.chriscool@tuxfamily.org","threadId":"17580","inReplyTo":"20090205141336.GA28443@elte.hu","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2009-02-06T05:23:15Z","receivedAt":"2009-02-06T05:23:15Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Le jeudi 5 février 2009, Ingo Molnar a écrit :\n> * Christian Couder wrote:\n> > Hi,\n> >\n> > For information, an article from me, 'Fully automated bisecting with\n> > \"git bisect run\"' has been published in today's edition of LWN on the\n> > development page:\n> >\n> > http://lwn.net/Articles/317154/\n>\n> Nice article!\n>\n> In terms of possible future enhancements of git bisect, here's a couple\n> of random ideas that would help my auto-bisection efforts:\n>\n> - Feature: support \"Bisection Redundancy\"\n>\n> This feature helps developers realize if a bug is sporadic. This\n> happens quite often in the kernel space: a bug looks deterministic, but\n> down the line it becomes sporadic. Sometimes a boot crash only occurs\n> with a 75% probability - and if one is unlucky it can cause a _lot_ of\n> wasted bisection time. The wrong commit gets blamed and the wrong set of\n> developers start scratching their heads. It's a reoccuring theme on lkml.\n>\n> What git could do here is to allow testers to inject a bit of extra\n> \"redundancy\" automatically, and use the redundant test-points to\n> detect conflicts in good/bad constraints.\n>\n> It would work like this:\n>\n> git bisect start --redundancy=33%\n>\n> It would mean that for every third bisection points, Git would\n> _not_ chose the ideal (estimated) 'middle point' from the set of\n> \"unknown quality\" changes that are still outstanding - but would\n> intentionally \"weer outside\" and select one commit from the _known_ set\n> of commits.\n>\n> If such a redundant re-test of the known-good or known-bad set yields\n> a nonsensical result then Git aborts the bisection with a \"logic\n> inconsistency detected\" kind of message - and people could at this point\n> realize the non-determinism of the test.\n>\n> ( Git can do this when a \"redundant\" test point is marked as 'bad' -\n> despite an earlier bisection already categorizing that test point as\n> 'good' - or if it's the other way around. Git will only continue\n> with the bisection if the test point has the expected quality. )\n>\n> This essentially means an automated re-test - but it's much better\n> than just a repeated bisection - i've often met non-deterministic bugs\n> that yield the _exact same_ nonsensical commit even on repeat bisections.\n> That happens when a timing bug depends on the exact kernel layout, or a\n> miscompilation or linker bug depends on the exact kernel layout, etc.\n>\n> It's also faster than a re-done bisection: 33% more testpoints is\n> better than twice as many test-points. Also, auto-bisection can deal with\n> redundancy just fine - it does not really matter whether i have to wait\n> 20 or 30 minutes for a test result since there's no manual intervention\n> needed - but it _very_ much matters whether i can trust the validity of\n> the bisection result.\n\nI see. With the current code it seems difficult to have a \"weer outside\" \nalgorithm, but if the \"git bisect skip\" code is ported (from shell \nin \"git-bisect.sh\") to C in \"builtin-rev-list.c\", that might be doable. And \nanyway moving the bisect skip code to C is needed for other improvements.\n\n> - Feature: better \"git bisect next\" support.\n\nYou probably mean \"git bisect skip\" here.\n\n> Sometimes a commit wont build. In that case we have \"git bisect next\",\n> but last i checked that only jumps a single commit - and build breakages \n> often have a large scope - full trees that got merged upstream, etc. Most\n> of the time those build breakages are uninteresting and the build-broken\n> window does not contain the bad commit.\n>\n> So it would be nice to have a \"git bisect next --left=20%\" type of\n> feature. This would jump 20% commits to the \"left\" from the bisection\n> point, towards the 'known bad' set of commits, but still within the\n> bisection window.\n>\n> Similarly, \"git bisect next --right=20%\" would jump towards the\n> known-good edge of the bisection window (but still within the bisection\n> window).\n\nIn the following thread, H. Peter Anvin suggested an algorithm to deal with \nthis kind of problem:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/98164/\n\nAnd I suggested a simpler one, that might be implemented without having to \nport \"git bisect skip\" code to C first, but I did not work on it yet.\n\n> Currently when i hit a build error during auto-bisection, it aborts and\n> i have to intervene manually. But with a bigger jump distance i could use\n> git-bisect-next reliably in scripts too.\n>\n> Likewise, users too hit build breakages often, and find it hard to get\n> out of the window of breakage. With the high-order tree structure of the\n> kernel repository that is rather non-intuitive to do as well, and often\n> people make mistakes and test the wrong commit.\n\nI am working slowly on \"git replace\" these days and, if everything goes \nwell, it should make it possible to use \"replace\" refs when bisecting, so \nthat people could bisect on commit trees where many breakages have been \nremoved. And as refs can be shared, this means that users and developers \nshould be able to easily share these improved trees.\n\nAnother way to work around breakages could be to have a list of commits and \nranges of commits that should always be skipped and always pass them \nto \"git bisect skip\" before using \"git bisect run\". Something like that \nperhaps:\n\n$ git bisect start \n$ git bisect skip $(cat always_skipped.txt)\n$ git bisect run ./my_test_script.sh\n\nBut I agree that it would also be a good thing to have an improved \"git \nbisect skip\" that could jump out of breakage windows.\n\n> - Feature: detect \"redundant\" and \"inconsistent\" test points\n>\n> This is a variation of the redunant testing theme, but from a different\n> angle: often newbies when they bisect the kernel weer outside of the\n> bisection window without realizing it. It would be nice if Git printed\n> a friendly notifier that:\n>\n> git bisect good 12341234\n> info: bisection point 12341234 was already in the 'good' range\n\nI agree that it would be nice. And it might be easy to implement.\n\n> Or, if the redunant test point is conflicting, print:\n>\n> git bisect good 12341234\n> fatal: bisection point 12341234 was already in the 'bad' range!\n>\n> And give an error return as well, so that scripts can abort.\n>\n> Currently Git seems to be very forgiving and accepts all bisection\n> points that we feed it, without checking them for consistency. (this\n> might have changed in current development versions, i dont know.)\n\nI don't think conflicting test points are accepted. An error should be \nreported (with exit code 1), like this:\n\n$ git bisect start HEAD~5 HEAD\nSome good revs are not ancestor of the bad rev.\ngit bisect cannot work properly in this case.\nMaybe you mistake good and bad revs?\n\n> - User friendliness: give an estimation about how many steps are\n> remaining\n>\n> Right now git prints this when a bisection session begins:\n>\n> aldebaran:~/tip> git bisect start\n> aldebaran:~/tip> git bisect bad linus\n> aldebaran:~/tip> git bisect good v2.6.28\n> Bisecting: 5449 revisions left to test after this\n> [e0b685d39a0404e7f87fb7b7808c3b37a115fe11] Updated contact info for\n> CREDITS file\n>\n> It would be nice if Git estimated the expected number of bisection\n> points. Something like this would be helpful:\n>\n> aldebaran:~/tip> git bisect good v2.6.28\n> Bisecting: 5449 revisions left to test after this\n> About ~16 test steps left [approximated]\n> [e0b685d39a0404e7f87fb7b7808c3b37a115fe11] Updated contact info for\n> CREDITS file\n>\n> The real number of test points might be higher than this, if the tree\n> layout is unlucky, or it might be less than this if the user manually\n> narrows the bisection window to a suspected set of commits - but that's\n> OK - most kernel testers use the default variant and the message is\n> clear enough that it's only an estimation.\n\nI will have a look at that. It might be easy to do.\n\nThanks,\nChristian.\n"},{"id":"103579","messageId":"200902070541.29955.chriscool@tuxfamily.org","threadId":"17580","inReplyTo":"200902060623.16046.chriscool@tuxfamily.org","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2009-02-07T04:41:29Z","receivedAt":"2009-02-07T04:41:29Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Le vendredi 6 février 2009, Christian Couder a écrit :\n> Le jeudi 5 février 2009, Ingo Molnar a écrit :\n>\n> > - Feature: better \"git bisect next\" support.\n>\n> You probably mean \"git bisect skip\" here.\n>\n> > Sometimes a commit wont build. In that case we have \"git bisect\n> > next\", but last i checked that only jumps a single commit - and build\n> > breakages often have a large scope - full trees that got merged\n> > upstream, etc. Most of the time those build breakages are uninteresting\n> > and the build-broken window does not contain the bad commit.\n> >\n> > So it would be nice to have a \"git bisect next --left=20%\" type of\n> > feature. This would jump 20% commits to the \"left\" from the bisection\n> > point, towards the 'known bad' set of commits, but still within the\n> > bisection window.\n> >\n> > Similarly, \"git bisect next --right=20%\" would jump towards the\n> > known-good edge of the bisection window (but still within the bisection\n> > window).\n>\n> In the following thread, H. Peter Anvin suggested an algorithm to deal\n> with this kind of problem:\n>\n> http://thread.gmane.org/gmane.comp.version-control.git/98164/\n>\n> And I suggested a simpler one, that might be implemented without having\n> to port \"git bisect skip\" code to C first, but I did not work on it yet.\n>\n> > Currently when i hit a build error during auto-bisection, it aborts\n> > and i have to intervene manually. But with a bigger jump distance i\n> > could use git-bisect-next reliably in scripts too.\n> >\n> > Likewise, users too hit build breakages often, and find it hard to\n> > get out of the window of breakage. With the high-order tree structure\n> > of the kernel repository that is rather non-intuitive to do as well,\n> > and often people make mistakes and test the wrong commit.\n>\n> I am working slowly on \"git replace\" these days and, if everything goes\n> well, it should make it possible to use \"replace\" refs when bisecting, so\n> that people could bisect on commit trees where many breakages have been\n> removed. And as refs can be shared, this means that users and developers\n> should be able to easily share these improved trees.\n>\n> Another way to work around breakages could be to have a list of commits\n> and ranges of commits that should always be skipped and always pass them\n> to \"git bisect skip\" before using \"git bisect run\". Something like that\n> perhaps:\n>\n> $ git bisect start \n> $ git bisect skip $(cat always_skipped.txt)\n> $ git bisect run ./my_test_script.sh\n\nIt might be useful to have a list of always good commits too, and use it \nlike this:\n\n$ git bisect start $(cat always_good.txt)\n$ git bisect skip $(cat always_skipped.txt)\n$ git bisect run ./my_test_script.sh\n\nor you may want to use grafts in your bisection repository. See the \nfollowing thread about bisection breakage in the btrfs history:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/105186/\n\nRegards,\nChristian.\n"},{"id":"103616","messageId":"ee77f5c20902070455h360d8476re76294735673b4ca@mail.gmail.com","threadId":"17580","inReplyTo":"200902070541.29955.chriscool@tuxfamily.org","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"David Symonds","fromEmail":"dsymonds@gmail.com","sentAt":"2009-02-07T12:55:34Z","receivedAt":"2009-02-07T12:55:34Z","isPatch":false,"sender":{"key":"dsymonds@gmail.com","avatar":"https://gravatar.com/avatar/b22f5051cbfc11836e36cf7a690e6cde4e225d835e13295ff98d15c7a9ee3c0f?d=mp&s=160"},"body":"On Sat, Feb 7, 2009 at 3:41 PM, Christian Couder\n wrote:\n\n> It might be useful to have a list of always good commits too, and use it\n> like this:\n>\n> $ git bisect start $(cat always_good.txt)\n> $ git bisect skip $(cat always_skipped.txt)\n> $ git bisect run ./my_test_script.sh\n\nYour test script could just do this at its start instead:\n\n if cat always_good.txt | grep $(rev-parse HEAD); then\n exit 0\n elif cat always_skipped.txt | grep $(rev-parse HEAD); then\n exit 125\n fi\n\n\nDave.\n"},{"id":"103640","messageId":"200902071909.14752.chriscool@tuxfamily.org","threadId":"17580","inReplyTo":"ee77f5c20902070455h360d8476re76294735673b4ca@mail.gmail.com","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2009-02-07T18:09:14Z","receivedAt":"2009-02-07T18:09:14Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Le samedi 7 février 2009, David Symonds a écrit :\n> On Sat, Feb 7, 2009 at 3:41 PM, Christian Couder\n>\n> wrote:\n> > It might be useful to have a list of always good commits too, and use\n> > it like this:\n> >\n> > $ git bisect start $(cat always_good.txt)\n> > $ git bisect skip $(cat always_skipped.txt)\n> > $ git bisect run ./my_test_script.sh\n>\n> Your test script could just do this at its start instead:\n>\n> if cat always_good.txt | grep $(rev-parse HEAD); then\n> exit 0\n\nThis won't work, because when you give one good rev, you say that all the \nancestors of this rev are good (not just that the rev you give is good).\n\n> elif cat always_skipped.txt | grep $(rev-parse HEAD); then\n> exit 125\n> fi\n\nIf you have a range like v1..v2 in \"always_skipped.txt\" this won't work.\nAnd anyway it may be less efficient because it may perform many useless \ncheckouts.\n\nRegards,\nChristian.\n"},{"id":"103641","messageId":"7vskmqw1s4.fsf@gitster.siamese.dyndns.org","threadId":"17580","inReplyTo":"ee77f5c20902070455h360d8476re76294735673b4ca@mail.gmail.com","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-07T18:16:27Z","receivedAt":"2009-02-07T18:16:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Symonds writes:\n\n> On Sat, Feb 7, 2009 at 3:41 PM, Christian Couder\n> wrote:\n>\n>> It might be useful to have a list of always good commits too, and use it\n>> like this:\n>>\n>> $ git bisect start $(cat always_good.txt)\n>> $ git bisect skip $(cat always_skipped.txt)\n>> $ git bisect run ./my_test_script.sh\n>\n> Your test script could just do this at its start instead:\n>\n> if cat always_good.txt | grep $(rev-parse HEAD); then\n> exit 0\n> elif cat always_skipped.txt | grep $(rev-parse HEAD); then\n> exit 125\n> fi\n\nDon't cat a file into grep, please.\n"},{"id":"103834","messageId":"20090209121943.GG17782@elte.hu","threadId":"17580","inReplyTo":"7vskmqw1s4.fsf@gitster.siamese.dyndns.org","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Ingo Molnar","fromEmail":"mingo@elte.hu","sentAt":"2009-02-09T12:19:43Z","receivedAt":"2009-02-09T12:19:43Z","isPatch":false,"sender":{"key":"mingo@elte.hu","avatar":null},"body":"\n* Junio C Hamano wrote:\n\n> David Symonds writes:\n> \n> > On Sat, Feb 7, 2009 at 3:41 PM, Christian Couder\n> > wrote:\n> >\n> >> It might be useful to have a list of always good commits too, and use it\n> >> like this:\n> >>\n> >> $ git bisect start $(cat always_good.txt)\n> >> $ git bisect skip $(cat always_skipped.txt)\n> >> $ git bisect run ./my_test_script.sh\n> >\n> > Your test script could just do this at its start instead:\n> >\n> > if cat always_good.txt | grep $(rev-parse HEAD); then\n> > exit 0\n> > elif cat always_skipped.txt | grep $(rev-parse HEAD); then\n> > exit 125\n> > fi\n> \n> Don't cat a file into grep, please.\n\nI do it all the time not because i dont know about grep's ability\nto take a file parameter, but because this way it's just a\nspecial-case of command piping and i can inject other commands as\ni extend/edit the command line interactively, etc.\n\n\tIngo\n"},{"id":"103836","messageId":"alpine.DEB.1.00.0902091414310.10279@pacific.mpi-cbg.de","threadId":"17580","inReplyTo":"20090209121943.GG17782@elte.hu","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-02-09T13:15:29Z","receivedAt":"2009-02-09T13:15:29Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 9 Feb 2009, Ingo Molnar wrote:\n\n> * Junio C Hamano wrote:\n> \n> > David Symonds writes:\n> > \n> > > On Sat, Feb 7, 2009 at 3:41 PM, Christian Couder\n> > > wrote:\n> > >\n> > >> It might be useful to have a list of always good commits too, and use it\n> > >> like this:\n> > >>\n> > >> $ git bisect start $(cat always_good.txt)\n> > >> $ git bisect skip $(cat always_skipped.txt)\n> > >> $ git bisect run ./my_test_script.sh\n> > >\n> > > Your test script could just do this at its start instead:\n> > >\n> > > if cat always_good.txt | grep $(rev-parse HEAD); then\n> > > exit 0\n> > > elif cat always_skipped.txt | grep $(rev-parse HEAD); then\n> > > exit 125\n> > > fi\n> > \n> > Don't cat a file into grep, please.\n> \n> I do it all the time not because i dont know about grep's ability\n> to take a file parameter, but because this way it's just a\n> special-case of command piping and i can inject other commands as\n> i extend/edit the command line interactively, etc.\n\nI think Junio meant using '< $file' type redirection, to avoid an \nunnecessary fork(). (Good habit, avoiding fork()s...)\n\nCiao,\nDscho\n"},{"id":"103897","messageId":"ee77f5c20902091303v1d268761ufdc85e2364097d84@mail.gmail.com","threadId":"17580","inReplyTo":"alpine.DEB.1.00.0902091414310.10279@pacific.mpi-cbg.de","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"David Symonds","fromEmail":"dsymonds@gmail.com","sentAt":"2009-02-09T21:03:23Z","receivedAt":"2009-02-09T21:03:23Z","isPatch":false,"sender":{"key":"dsymonds@gmail.com","avatar":"https://gravatar.com/avatar/b22f5051cbfc11836e36cf7a690e6cde4e225d835e13295ff98d15c7a9ee3c0f?d=mp&s=160"},"body":"On Tue, Feb 10, 2009 at 12:15 AM, Johannes Schindelin\n wrote:\n\n> I think Junio meant using '< $file' type redirection, to avoid an\n> unnecessary fork(). (Good habit, avoiding fork()s...)\n\nYes, I usually pass filenames to grep; this time I was\ncopy-and-pasting parts of the original script. But my point still\nremains: it seems it would be cleaner to do this kind of\nalways-good-filtering in the test script rather than make git-bisect\nmore complex.\n\n\nDave.\n"},{"id":"103950","messageId":"200902100712.31820.chriscool@tuxfamily.org","threadId":"17580","inReplyTo":"ee77f5c20902091303v1d268761ufdc85e2364097d84@mail.gmail.com","subject":"Re: Article about \"git bisect run\" on LWN","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2009-02-10T06:12:31Z","receivedAt":"2009-02-10T06:12:31Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Le lundi 9 février 2009, David Symonds a écrit :\n> On Tue, Feb 10, 2009 at 12:15 AM, Johannes Schindelin\n>\n> wrote:\n> > I think Junio meant using '< $file' type redirection, to avoid an\n> > unnecessary fork(). (Good habit, avoiding fork()s...)\n>\n> Yes, I usually pass filenames to grep; this time I was\n> copy-and-pasting parts of the original script. But my point still\n> remains: it seems it would be cleaner to do this kind of\n> always-good-filtering in the test script rather than make git-bisect\n> more complex.\n\nBut it's not so easy and not efficient to do this kind of filtering in the \ntest script.\n\nMaybe something cleaner could be to have \"always good\" and \"always skipped\" \nrefs that are automatically used each time you bisect.\n\nFor example, \"git bisect good --always\" and \"git bisect skip --always\" could \nadd refs in \"refs/bisect/always/\" even when you are not bisecting, and \nthese refs would not be removed on reset.\n\nBut in the end, I think this does not really fix the problem. If you need \nsuch hacks, it means that your commit DAG is not easily bisectable. For \nexample if you must skip too many commits, then too often bisection will \nnot be able to point to a first bad commit because it won't be able to tell \nbetween many skipped commits.\n\nThat's why I think the proper fix is to have a way to bisect on a fixed up \ncommit DAG where you can just get rid of old annoying bugs and of history \nparts that are not relevant.\n\nRegards,\nChristian.\n"}]}