Re: Reporting bugs and bisection

From: Willy Tarreau <w@1wt.eu>
Date: Apr 14, 2008, 04:39 UTC
Message-ID: <20080414043939.GA6862@1wt.eu>
In-Reply-To: <alpine.DEB.1.10.0804131546370.9318@asgard>

On Sun, Apr 13, 2008 at 04:51:34PM -0700, david@lang.hm wrote:

Show 36 quoted lines

> cross-posted to git for the suggestion at the bottom
> 
> On Sun, 13 Apr 2008, Stephen Clark wrote:
> 
> >Evgeniy Polyakov wrote:
> >>On Sun, Apr 13, 2008 at 10:33:49PM +0200, Rafael J. Wysocki (rjw@sisk.pl) 
> >>wrote:
> >>>Things like this are very disappointing and have a very negative impact 
> >>>on bug
> >>>reporters.  We should do our best to avoid them.
> >>
> >>Shit happens. This is a matter of either bug report or those who were in
> >>the copy list. There are different people and different situations, in
> >>which they do not reply.
> >>
> >Well less shit would happen if developers would take the time to at least 
> >test their patches before they were submitted. It like we will just have 
> >the poor user do our testing for us. What kind of testing do developers 
> >do. I been a linux user and have followed the LKML for a number of years 
> >and have yet to see
> >any test plans for any submitted patches.
> 
> I've been reading LKML for 11 years now, I've tested kernels and reported 
> a few bugs along the way.
> 
> the expectation is that the submitter should have tested the patches 
> before submitting them (where hardware allows). but that "where hardware 
> allows" is a big problem. so many issues are dependant on hardwre that 
> it's not possible to test everything.
> 
> there are people who download, compile and test the tree nightly (with 
> farms of machines to test different configs), but they can't catch 
> everything.
> 
> expecting the patches to be tested to the point where there are no bugs is 
> unreasonable.

[...]

Agreed. The difficulty is that only the developer knows how confident he is in his code. Even the subsystem maintainer does not know, which is the real issue since as long as the code is not identified, he does not know whom to ping.

And I think that it might help if we could add a "Trust" rating to the patches we submit, similarly to "Tested-By" or "Signed-off-by". We could use 1 to 5. Basically, when the patch was completed at 3am and just builds, it's more likely 1/5. When it has been stressed for 1 week, it would be 4/5. 5/5 would only be used in backports of known working code, for some wide-used external patches, or for trivial patches (eg: doc/whitespace fixes). The goal would clearly not be to just trust patches with a high rate (since they might break when associated with others), but for the subsystem maintainer to quickly check if there are some of them the author does not 100% trust, in which case he could ping the author to check if his patch *may* cause the reported problem.

What makes this rating system delicate is that the rate cannot be changed afterwards. But after all, that's not much of a problem. A bug may very well reveal itself one year after the code was merged, so it's really the developer's estimation which matters.

For this to be efficiently used, we would need git-commit to accept a new "-T <rating>" argument with the following possible values :

   0: untested (default)
   1: builds
   2: seems to be working
   3: passed basic non-regression tests
   4: survived stress testing at the developer's
   5: known to be working for a long time somewhere else

I'm sure many people would find this useless (or in fact reject the idea because it would show that most code will be rated 1 or 2), but I really think it can help subsystem maintainers make the relation between a reported bug and a possible submitter.

Willy