Re: AAARGH bisection is hard (Re: [2.6.39 regression] X locks up hard right after logging in)

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: May 13, 2011, 19:18 UTC
Message-ID: <BANLkTinGOHX-fNuQqZr50nvUC4BTymwBNg@mail.gmail.com>
In-Reply-To: <7vliya77xl.fsf@alter.siamese.dyndns.org>

On Fri, May 13, 2011 at 11:48 AM, Junio C Hamano <gitster@pobox.com> wrote:

Show 6 quoted lines

>
> Could you please clarify "off-limits"?
>
> Do you mean "anything before v2.6.38 did not even have this feature, so
> the result of testing a version in that range does not give us any
> information"?

Well, I think it's useful in two cases.

It's useful for the "before this version, the test we're doing doesn't even make sense and cannot succeed" sense.

That doesn't have to be about hardware support, it could be any feature. For example, in git, say that you noticed that --dirstat-by-file stopped working at some point. You know it was good when you merged it, so you'd do

  git bisect start
  git bisect good ac9666f84a59

but you'd also go "that's also when I introduced the *test* for it, so I'll need to require that":

  git bisect requires ac9666f84a59

and then you can start it all off:

  git bisect bad
  git bisect run sh -c "make test"

or whatever.

Because you don't want to go into the merges that were based on code that didn't even _have_ that feature.

Ok, so that's a made-up and contrieved example (it would make more sense for when you add a whole new flag, and your test-script is testign that new functionality), but it kind of explains the notion: it will not bother to run bisect on code that simply isn't _relevant_ for the issue you are bisecting.

> Upon seeing "bad" result from a version before v2.6.38, what can we conclude?

The point would be that such versions aren't even _testable_. So the whole "seeing 'bad'" concept is a NULL concept. It's like the above "new command line flag to 'git'" example: it's not that those commits might not have broken something, but those commits are crazy to test.

If it turns out that a merge brought in the breakage, we'd have to do a totally new kind of test thing. But from a bisect standpoint, it's already very interesting if the end result is "hey, you merged that code that didn't even _support_ the feature we're testing, and that broke it". That gives quite a bit of information, and opens up new avenues for testing.

For example, at that point, we might decide that "Oh, ok, now I will need to re-run the bisect for everthing that came in in that merge, but I will do a new merge at that point to see which commit it is that doesn't play nice with the new feature".

> The breakage cannot possibly come from the feature
> that is being checked, so the procedure to check itself is busted?

Right.

HOWEVER.

There's another reason to say "require version XYZ", and that's essentially a "I want to do a (quicker) high-level bisect". Especially the way the kernel merge window is done, it might be that versions prior to v2.6.38 work perfectly _fine_, but what you want to do is to quickly bisect down to which subsystem caused breakage.

A good way to do that would be to just say "requires v2.6.38", and suddenly the actual set of commits that we're going to bisect is going to be *much* smaller. We're basically throwing away all the individual commits that were merged in the merge window, and saying something that approximates to "we are only interested in the merge points".

Why would we do that? Just to get a quicker "this is the problematic subsystem". So the "requires xyz" might be quite useful for that reason too.

                 Linus