Re: Reporting bugs and bisection

> On Mon, 14 Apr 2008 01:04:12 -0700
> > 
> > The steps to be taken are:
> > 
> > a) agree that we have a problem
> > 
> 
> 
> I for one do not agree that we have a problem.
> 
> Based on actual data on oopses (which very clearly excludes other kinds of bugs, so I know I only see part of the story)
> we are doing reasonably well. Lets look at the 2.6.25 cycle. 
> We got a total of roughly 2700 reports of oopses/warn_ons from users. (This may sound high to those of you only reading
> lkml, but this includes automatically collected oopses from Fedora 9 beta testers).
> Out of these 2700, the top 20 issues account for 75% of the total reports.
> 
> Out of these 20 issues, 9 were from still out of tree drivers (wireless.git and drm.git included in F9). These were
> caught before they even got close to mainline.
> The remaining 11 issues can be split in
> 1) The ones we caught and fixed
> 2) TCP/IP warnings that DaveM and co are chasing down hard (but have trouble finding reproducers)
> 3) An EXT3 bug that in theory can cause data corruption, but in practice seems to happen after you yank out a USB stick
>   with an EXT3 filesystem on (so it can't corrupt the disk data). Ted is working on this
> 4) A bug (double free) that hits in the skb layer, probably caused by a bug in the ipv4 code
>    (a first analysis + potential patch was mailed to netdev this weekend)
> 5) sysfs "existing file added" warning, mostly in the USB stack
>    (gregkh claims he fixed this recently, I'm not entirely sure he got all cases)
> 
> And when I look beyond the first 20, the same pattern arises, we fixed the majority of the issues before -rc9.
> At position 25 we have less than 20 reports per bug. At position 35 we have less than 10 reports per bug. 
> At position 50 we have less than 5 reports per bug. Conclusion there: the bugs people actually hit fall of dramatically;
> there's a core set of issues that gets hit a lot, the rest quickly gets reduced to noise levels.
> 
> 
> To me this does not sound like we have a huge quality problem because
> 1) The distribution of the bugs is such that there is a relatively small set of core issues
>    that are widely hit, and then there's a near exponential drop after that
> 2) We are fixing the important bugs by and large before they hit a release
>    (important as defined by the number of people actually hitting the bug)
> 
> 
>  
> I'll be writing a report with more details about this soon with more analysis and statistics
> (I'll be looking at more detail around the top 25 issues, when they got introduced, when they got fixed etc)