From: david@lang.hm
Date: Fri, 06 Feb 2009 02:42:16 GMT
Subject: Re: Article about "git bisect run" on LWN
Message-ID: <alpine.DEB.1.10.0902051838180.5340@asgard.lang.hm>
In-Reply-To: <20090205141336.GA28443@elte.hu>

On Thu, 5 Feb 2009, Ingo Molnar wrote:

> * Christian Couder <chriscool@tuxfamily.org> wrote:
>
>> Hi,
>>
>> For information, an article from me, 'Fully automated bisecting with "git
>> bisect run"' has been published in today's edition of LWN on the
>> development page:
>>
>> http://lwn.net/Articles/317154/
>
> Nice article!
>
> In terms of possible future enhancements of git bisect, here's a couple of
> random ideas that would help my auto-bisection efforts:
>
> - Feature: support "Bisection Redundancy"
>
>   This feature helps developers realize if a bug is sporadic. This happens
>   quite often in the kernel space: a bug looks deterministic, but down the
>   line it becomes sporadic. Sometimes a boot crash only occurs with a 75%
>   probability - and if one is unlucky it can cause a _lot_ of wasted
>   bisection time. The wrong commit gets blamed and the wrong set of
>   developers start scratching their heads. It's a reoccuring theme on lkml.
>
>   What git could do here is to allow testers to inject a bit of extra
>   "redundancy" automatically, and use the redundant test-points to detect
>   conflicts in good/bad constraints.
>
>   It would work like this:
>
>      git bisect start --redundancy=33%
>
>   It would mean that for every third bisection points, Git would
>   _not_ chose the ideal (estimated) 'middle point' from the set of "unknown
>   quality" changes that are still outstanding - but would intentionally
>   "weer outside" and select one commit from the _known_ set of commits.
>
>   If such a redundant re-test of the known-good or known-bad set yields a
>   nonsensical result then Git aborts the bisection with a "logic
>   inconsistency detected" kind of message - and people could at this point
>   realize the non-determinism of the test.
>
>   ( Git can do this when a "redundant" test point is marked as 'bad' -
>     despite an earlier bisection already categorizing that test point as
>     'good' - or if it's the other way around. Git will only continue with
>     the bisection if the test point has the expected quality. )
>
>   This essentially means an automated re-test - but it's much better than
>   just a repeated bisection - i've often met non-deterministic bugs that
>   yield the _exact same_ nonsensical commit even on repeat bisections. That
>   happens when a timing bug depends on the exact kernel layout, or a
>   miscompilation or linker bug depends on the exact kernel layout, etc.
>
>   It's also faster than a re-done bisection: 33% more testpoints is better
>   than twice as many test-points. Also, auto-bisection can deal with
>   redundancy just fine - it does not really matter whether i have to wait
>   20 or 30 minutes for a test result since there's no manual intervention
>   needed - but it _very_ much matters whether i can trust the validity of
>   the bisection result.

when you gave this the title of redundnancy and described the problem I 
assumed that you would then propose running the test multiple times (so 
"git bisect run X --redundancy 5" would run each test 5 times, it would 
pass IFF it passed the test all 5 times. that would seem to be a better 
match for the name, as well as being a better test

David Lang