git-bisect problem

I've been trying to locate an ipw2200 regression in Jeff's tree (git+ssh://master.kernel.org/pub/scm/linux/kernel/git/jgarzik/netdev-2.6.git#ALL) and it ended up leading me to

826eeb53a6f264842200d3311d69107d2eb25f5e is first bad commit
diff-tree 826eeb53a6f264842200d3311d69107d2eb25f5e (from 33052057e3e2db7f37fc78aa3f25c98f7e989fae)
Author: Linus Torvalds <torvalds@g5.osdl.org>
Date:   Thu Feb 2 22:03:08 2006 -0800

    Linux v2.6.16-rc2

which wasn't very useful.

I don't _think_ I screwed anything up.

git-bisect start # good: [2664b25051f7ab96b22b199aa2f5ef6a949a4296] Linux v2.6.16-rc1 git-bisect good 2664b25051f7ab96b22b199aa2f5ef6a949a4296 # bad: [826eeb53a6f264842200d3311d69107d2eb25f5e] Linux v2.6.16-rc2 git-bisect bad 826eeb53a6f264842200d3311d69107d2eb25f5e # good: [10379a25fee8ddc8698d2f6c54ccedd4664c2941] Merge master.kernel.org:/pub/scm/linux/kernel/git/davej/agpgart git-bisect good 10379a25fee8ddc8698d2f6c54ccedd4664c2941 # good: [9a2dba4b4912b493070cbc170629fdbf440b01d7] slab: rename ac_data to cpu_cache_get git-bisect good 9a2dba4b4912b493070cbc170629fdbf440b01d7 # good: [9ad11ab48b1ad618bf47076e9e579f267f5306c2] compat: fix compat_sys_openat and friends git-bisect good 9ad11ab48b1ad618bf47076e9e579f267f5306c2 # good: [1494a92f4c2b1d5abdaa1f823dd19f797bb137de] [ALSA] hda-codec - Fix typos in alc882 model table git-bisect good 1494a92f4c2b1d5abdaa1f823dd19f797bb137de # good: [9fdb62af92c741addbea15545f214a6e89460865] [ACPI] merge 3549 4320 4485 4588 4980 5483 5651 acpica asus fops pnpacpi branches into release git-bisect good 9fdb62af92c741addbea15545f214a6e89460865 # good: [cf41f8ac386e8d62122e7e394b4c6b3e3ab30ede] Merge branch 'drm-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6 git-bisect good cf41f8ac386e8d62122e7e394b4c6b3e3ab30ede # good: [00b464debf0038b1628996065f0be564ccfbfd86] SUNRPC: Remove obsolete rpcauth #defines git-bisect good 00b464debf0038b1628996065f0be564ccfbfd86 # good: [35849c75d7750a254119c1a4b88c90156919df2a] md: Add sysfs access to raid6 stripe cache size git-bisect good 35849c75d7750a254119c1a4b88c90156919df2a # good: [33052057e3e2db7f37fc78aa3f25c98f7e989fae] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6 git-bisect good 33052057e3e2db7f37fc78aa3f25c98f7e989fae

Andrew Morton <akpm@osdl.org> writes:

At this point, looking at "git bisect visualize" shows that bisect point is at "SUNRPC: Remove obsolete rpcauth #defines", and commits older than that are NFSv3, 4 SUNRPCs, 2 NLMs, and stops at "[PATCH] kernel-doc: clean up the script (whitespace)".

And this is marked to be good -- it leaves:

       SUNPRC    good
   SUNRPC  NFSv3 00b464
  ...o---o---o---o---------o--------o
                          /         bad
     o---o---o---o---o---o          v2.6.16-rc2
 good  
 cf41f8  md  md  md  md  dm

Then you mark the rightmost md to be good.

                 good
                 00b464
                 o---------o--------o
                          /         bad
                     o---o          v2.6.16-rc2
                   good
                     md  dm

So at this point, assuming the bug is something that is bisectable, there are still three suspects:

  (1) dm (device-mapper log bitset: fix big endian)
  (2) the merge was screwed up 
  (3) Linus did more than setting EXTRAVERSION in v2.6.16-rc2

And your test showed the merge one was good.

                          good
                          330520
                 o---------o--------o
                          /         bad
                     o---o          v2.6.16-rc2
                   good
                     md  dm

As humans, we can tell that it is not very plausible that the EXTRAVERSION change caused whatever breakage you are chasing, but sorry, from your log, I think bisect is doing the right thing.

The last stretch of the md/dm track does not seem to have much to do with ipw2200 (isn't that a wireless thing?), and the other track does not look card specific even though NFS and SUNRPC sounds networking related. If I have to guess:

 (0) the bug is not really reproducible;

 (1) an earlier part of bisection misrecorded bad as good;

 (2) older commits on these two tracks have subtle interaction,
     and the problem does not surface without such interaction
     (but that is not plausible because your test on the final
     "merge" should have shown the problem if that is the case);

Junio C Hamano <junkio@cox.net> wrote:

I don't think humans are well-suited to using git.

My current theory is that I was bisecting Linus's tree all along.

What is the correct way in which to switch to git-netdev-all in preparation for performing the bisection?

On Mon, Feb 13, 2006 at 01:32:05AM -0800, Andrew Morton wrote:

First, use "git branch" to show you what branches exist, the * will mark the current one.

Then "git checkout $branch" to switch to one that exists, or "git checkout -b $newbranch $sourcebranch" to create a new branch starting from $sourcebranch (which can also be a random commit/tag/etc).

Ryan Anderson <ryan@michonline.com> wrote:

Yeah, am (ret)trying that.

Assuming I find the bad commit, how do I extract it as a patch?

I tried

git-format-patch -o ~/a 386093ef9a6c88576d8b418bf1c8616d5e410a20 git-netdev-all

and that chewed 10 minutes CPU time and produced no output, so I killed it.

On Mon, Feb 13, 2006 at 01:51:46AM -0800, Andrew Morton wrote:
| Assuming I find the bad commit, how do I extract it as a patch?
| 
| I tried
| 
| git-format-patch -o ~/a 386093ef9a6c88576d8b418bf1c8616d5e410a20 git-netdev-all
| 
| and that chewed 10 minutes CPU time and produced no output, so I killed it.

This gives you the whole info about the commit, including a patch:

git cat-file commit 386093ef9a6c88576d8b418bf1c8616d5e410a20

Cheers, Ferdy

--- "Fernando J. Pereda" <ferdy@ferdyx.org> wrote:

I personally like

   git-diff-file

--- "Fernando J. Pereda" <ferdy@ferdyx.org> wrote:

I meant to say:

   git-diff-tree --pretty -p <commit_id>

Luben

On Mon, Feb 13, 2006 at 10:58:59AM +0100, Fernando J. Pereda wrote:

Eh?  Then why not call the command "cat-object" or simply "cat"?

On Mon, Feb 13, 2006 at 01:51:46AM -0800, Andrew Morton wrote:

Well, assuming it's not a merge, you'll want something like this:

git format-patch -o ~/a 386093ef9a6c88576d8b418bf1c8616d5e410a20^1..386093ef9a6c88576d8b418bf1c8616d5e410a20

For essentially the same output, you can do a few other variations:

git whatchanged -p 386093ef9a6c88576d8b418bf1c8616d5e410a20^1..386093ef9a6c88576d8b418bf1c8616d5e410a20 git diff 386093ef9a6c88576d8b418bf1c8616d5e410a20^1..386093ef9a6c88576d8b418bf1c8616d5e410a20

If it's a merge that bisect terminates on, things get a bit trickier, as
you want to figure out what went wrong in the merge to cause it, so
you'll want to use either the syntax for specifying which merge parent
to look at (which I forget at the moment) or, run:
	git rev-list --parents --max-count=1 386093ef9a6c88576d8b418bf1c8616d5e410a20
and look at columns 2+ individually.

In fact, if you want, you can re-do the merge, by creating some branches based off of each parent, then pulling one into the other, and seeing what went wrong.

Hope that helps (if not, I apologize - I should've gone to bed a while ago and it may have snuck through)

Ryan Anderson <ryan@michonline.com> wrote:

That worked.

It did terminate on a merge. Thats over four hours gone and, frankly, I'm sick of it. I just want the darned diffs so I can do something useful.

It does.

I'm still not having much success geting a string of patches out of it.

git format-patch -o ~/a d834a41c966c6a20368fadb59248740935e6fbae..826eeb53a6f264842200d3311d69107d2eb25f5e

Has chewed 5 minutes CPU so far and hasn't produced anything.

How do I get the IPW patches out of Jeff's tree, in order?

I guess since I found a command which actually works, I can type that 20-odd times.

On Mon, 13 Feb 2006, Andrew Morton wrote:

Well, really, it's much nicer these days to just say

	git show 386093ef9

and you're done.

For me, it gives a nice

	diff-tree 386093ef9a6c88576d8b418bf1c8616d5e410a20 (from ce5f8d70ba6e3d7ffcaff86b2cf91a42c27f77af)
	Author: Alexey Dobriyan <adobriyan@gmail.com>
	Date:   Wed Feb 1 03:04:57 2006 -0800
	
	    [PATCH] ipw2200: fix ->eeprom[EEPROM_VERSION] check
	    
	    priv->eeprom is a pointer.
	    
	    Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
	    Acked-by: Yi Zhu <yi.zhu@intel.com>
	    Cc: James Ketrenos <jketreno@linux.intel.com>
	    Signed-off-by: Andrew Morton <akpm@osdl.org>
	    Signed-off-by: Linus Torvalds <torvalds@osdl.org>
	
	diff --git a/drivers/net/wireless/ipw2200.c b/drivers/net/wireless/ipw2200.c
	index 916b24c..14beab4 100644
	--- a/drivers/net/wireless/ipw2200.c
	+++ b/drivers/net/wireless/ipw2200.c
	@@ -2456,7 +2456,7 @@ static void ipw_eeprom_init_sram(struct 
	           copy.  Otherwise let the firmware know to perform the operation
	           on it's own
	         */
	-       if ((priv->eeprom + EEPROM_VERSION) != 0) {
	+       if (priv->eeprom[EEPROM_VERSION] != 0) {
	                IPW_DEBUG_INFO("Writing EEPROM data into SRAM\n");
	 
	                /* write the eeprom data to sram */

which looks sane.

well, that's 1003 patches you're asking for. 

That's almost certainly _not_ what you want.

Do "gitk ..args.." to visually see what you're doing. Or, what I did:

	git-rev-list d834a41c966c6a20368fadb59248740935e6fbae..826eeb53a6f264842200d3311d69107d2eb25f5e |
		wc -l

which is how I got the 1003.

I'm pretty sure it wasn't what you meant to do.

"git show". Much simpler. HOWEVER. Doing that 20-odd times sounds insane. Just use gitk to see that you actually have the right starting and ending points.

Visualizing the history really is very important. If you had tried gitk, you'd have immediately seen what you were doing, and that it wasn't what you wanted.

Only after you're really comfortable with git should you do anything at all without looking at it visually first.

(After you've done that a few months, you won't need it any more - your brain will be able to visualize things on its own. gitk is just the training wheels).

			Linus

Andrew,

Here is the output:

$ git-diff-tree --pretty -p 386093ef9a6c88576d8b418bf1c8616d5e410a20
diff-tree 386093ef9a6c88576d8b418bf1c8616d5e410a20 (from ce5f8d70ba6e3d7ffcaff86b2cf91a42c27f77af)
Author: Alexey Dobriyan <adobriyan@gmail.com>
Date:   Wed Feb 1 03:04:57 2006 -0800

    [PATCH] ipw2200: fix ->eeprom[EEPROM_VERSION] check
    
    priv->eeprom is a pointer.
    
    Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
    Acked-by: Yi Zhu <yi.zhu@intel.com>
    Cc: James Ketrenos <jketreno@linux.intel.com>
    Signed-off-by: Andrew Morton <akpm@osdl.org>
    Signed-off-by: Linus Torvalds <torvalds@osdl.org>

Luben Tuikov <ltuikov@yahoo.com> wrote:

Yes, that is decent. But for many patches, I'd end up having to call the files "386093ef9a6c88576d8b418bf1c8616d5e410a20.patch". git-format-patch chooses nice filenames. Slowly.

Anyway, repeated applications of the one-diff git-format-patch (based on a grep of the git-log output) got me the 69 patches which I want, so I can find this bug now, thanks.

Andrew Morton <akpm@osdl.org> writes:

A single commit is either:

	git format-patch -o ~/a 386093^ 386093
	git show 386093

But if you _did_ want to get everything that builds on top of 386093 (and Linus counted 1000+ commits if I recall), format-patch could be optimized. It currently does a lot more than just format 1000+ commits, to handle case where "his" and "mine" are not linear history and may have the same change acquired by applying the same patch:

          1---2---3 mine
         /
     ---4---5---6 his

In this picture, it does not just format 1 2 3. It first checks 1 2 3 5 6, and if each of 1 2 3 introduces the same change as either 5 or 6 introduces to omit it from the output. If 2 and 5 are the same change from 1 and 4 respectively, the final result has 1 and 3. This is OK and useful for smaller branch, but clearly expensive for long branches.

This is omitted when the ancestry graph would look like this:
	
          1---2---3 mine
         /
     ---4 his

but that would not have helped in this case anyway.

Maybe we could have --no-omit-common flag or something to disable this check.

Andrew Morton <akpm@osdl.org> writes:

I did not mean that ;-).  Git is not as smart as humans.

Sorry, I did not realize that was _not_ what you were doing. Your log started by saying 2.6.16-rc1 is good but 2.6.16-rc2 was not, so I just assumed your bug was between those two.

If your suspect was merged between these two versions, then it does not matter which branch you were _on_ when you started to bisect.

You mark points that are good and bad, and wander around in the commit DAG, trying to narrow down the distance between known good points and bad points while bisecting, and during that, you are not really on _any_ branch.

Junio C Hamano <junkio@cox.net> wrote:

So how am I supposed to find this bug in Jeff's tree?

I do git-checkout -f git-netdev-all, then do the bisection and I come up with junk.

<does it all again>

It points at this:

commit a03b1950521466e007288a25c9fc7ac7f05a97e5
Merge: 0b310f36d7d96e27f6941ec0f9b95e15142f1e78 c6f0d75a2defe8c7d8bf9f78de891cedc46b4b3e
Author: Jeff Garzik <jgarzik@pobox.com>
Date:   Tue Jan 31 11:52:21 2006 -0500

    Merge branch 'upstream-fixes'

git-bisect start # good: [d834a41c966c6a20368fadb59248740935e6fbae] ipw2200: do not sleep in ipw_request_direct_scan git-bisect good d834a41c966c6a20368fadb59248740935e6fbae # bad: [b0afb58735e5dae05cb06ce6d0ca3073f390e9dc] Merge branch 'upstream' git-bisect bad b0afb58735e5dae05cb06ce6d0ca3073f390e9dc # good: [0c19585b0d2f6817dd9af607650d3f6cae2fd8bc] uml: typo fixup git-bisect good 0c19585b0d2f6817dd9af607650d3f6cae2fd8bc # good: [71baa1a599c04ab56ebf5fdb8d03abd0d601462f] [MIPS] Get rid of unnecessary prototypes. Fixes and optimizations for HZ > 100. git-bisect good 71baa1a599c04ab56ebf5fdb8d03abd0d601462f # good: [d04e4e115bd9df2b748cb30abd610f3c0eb1e303] eeh_driver NULL noise removal git-bisect good d04e4e115bd9df2b748cb30abd610f3c0eb1e303 # good: [9908104935325bd6beba67d637b6f5396d47075c] [IPV6]: Address autoconfiguration does not work after device down/up cycle git-bisect good 9908104935325bd6beba67d637b6f5396d47075c # good: [0b310f36d7d96e27f6941ec0f9b95e15142f1e78] Merge branch 'upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-2.6 git-bisect good 0b310f36d7d96e27f6941ec0f9b95e15142f1e78 # bad: [70c07e02625ec46d0ffbfce1acef42d660803528] Merge branch 'viro' git-bisect bad 70c07e02625ec46d0ffbfce1acef42d660803528 # good: [2746b8623abce815aaae7afc946b1b39f8436f5a] Merge branch 'net.b0' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/bird git-bisect good 2746b8623abce815aaae7afc946b1b39f8436f5a # bad: [6bd0e10e53cc4824cd8cdaab8c370e53ab2e23c2] Merge branch 'sundance' git-bisect bad 6bd0e10e53cc4824cd8cdaab8c370e53ab2e23c2 # bad: [3c9b3a8575b4f2551e3b5b74ffa1c3559c6338eb] Merge branch 'master' git-bisect bad 3c9b3a8575b4f2551e3b5b74ffa1c3559c6338eb # bad: [c0d3c0c0ce94d3db893577ae98e64414d92e49d8] [netdrvr] schedule eepro100 for removal git-bisect bad c0d3c0c0ce94d3db893577ae98e64414d92e49d8 # bad: [a03b1950521466e007288a25c9fc7ac7f05a97e5] Merge branch 'upstream-fixes' git-bisect bad a03b1950521466e007288a25c9fc7ac7f05a97e5

Andrew Morton <akpm@osdl.org> writes:

Sorry, this question is what I do not quite understand.

Here is my understanding of the situation.

 - Betweeen 2.6.16-rc1 and 2.6.16-rc2 a bug you are chasing was
   introduced.  You know rc1 works fine but rc2 is bad.

 - You suspect that changes introduced by merging Jeff's tree
   at some point between -rc1 and -rc2 may be causing this.

Am I totally misunderstanding the situation?

Bisecting, starting from -rc1 and -rc2 marked good and bad would find the bug provided if the symptom is caused by a single bug (that is, before that commit things work but after that things stop working) that is in any commit that was not present in -rc1 but in -rc2. That includes what was merged from Jeff's tree, so even if you were "bisecting Linus' tree all along", if -rc1 was good and -rc2 was bad, that would have found the bug in Jeff's tree (if it was what introduced the bug). As long as that was merged between these two -- but otherwise breakage in -rc2 would not have anything to do with Jeff's tree, so that is one reason I am confused by your emphasis on "in Jeff's tree" part.

              good                     bad
              -rc1                     -rc2
    ---o---o---o---*---*---*---*---*---*
        \           \     /   /
         \           *---*   /
          \                 / 
           *---*---*---*---*
                Jeff's       

Your bisection that starts with good -rc1 and bad -rc2 would try to bisect commits that are not reachable from good ones (that is, parents before good ones are assumed to be good, since bisect is only good to look for a single regression), so at the beginning, all '*' commits are suspects. bisection picks one of them and after testing it, depending on it is good or bad, mark about the half of the remaining graph "unsuspected". So even if you start out with two commits on Linus' tree, you will wander into Jeff's tree, if the suspect commit is in there. I am confused by your emphasis on "in Jeff's tree" part.

Maybe you are saying that you _know_ what broke was from Jeff's tree, and felt bisecting other parts of Linus' tree was wasteful?

If so, you can use gitk to visualize the graph, find the merge Linus' did to merge from Jeff (and I presume you already did so and that is why you are suspecting Jeff's tree). Mark that as bad, not -rc2. Also if you know the commit on Linus' tree before that merge was good (and I presume you already did so and that is why you are suspecting Jeff's tree), mark that as good, not -rc1.

              -rc1      good  bad     -rc2
    ---o---o---o---o---o---o---*---o---o
        \           \     /   /
         \           o---o   /
          \                 / 
           *---*---*---*---*
                Jeff's       

Then your bisect will walk over commits on Jeff's tree.

Is this helpful, or am I still completely useless?

Junio C Hamano <junkio@cox.net> wrote:

yup ;)

The bug is in Jeff's tree only (git+ssh://master.kernel.org/pub/scm/linux/kernel/git/jgarzik/netdev-2.6.git#ALL) so I wanted to perform the bisection on the git-netdev-all branch.

So I did a `git log git-netdev-all' and looked at where the ipw2200 changes were and then decided that the 2.6.16-rc1 and 2.6.16-rc2 commits straddled those changes nicely, so I chose those as the bisection starting points.

On Mon, 13 Feb 2006, Andrew Morton wrote:

Actually, what you should do, is not play any games at all, but just tell "git bisect" what the problem is. It will do the right thing.

So in this case, what you do is _literally_ to

	# fetch Jeff's tree (you obviously had this already, but I just 
	# want to point it out as a "name that branch" thing)
	git fetch netdev-all

	# we know that that tree is broken
	git bisect start
	git bisect bad netdev-all

	# We know that Linus' top-of-tree doesn't have the bug
	git bisect good origin

and off you go. It absolutely magically does the right thing, and will bisect stuff that is only in the netdev branch and not in my tree. No guessing necessary, no need to try to figure out what the differences are. git will do it all for you.

And notice how it will work perfectly well, even if the two points you have tested AREN'T EVEN DIRECTLY RELATED! The "good" and "bad" points do not have to have any direct relationship other than a common parent _somewhere_. "git bisect" really is that good.

(The above is obviously assuming that "origin" is set to my tree, self-aggrandizing bastard that I am, and that you've set up a .git/remotes/netdev-all file pointing to Jeff's tree - your setup may vary from this, so you'd have to change the lines to match)

			Linus

Dear diary, on Tue, Feb 14, 2006 at 01:56:20AM CET, I got a letter where Andrew Morton <akpm@osdl.org> said that...

But aren't those commits on the Linus' "branch", not containing any commits specific to git-netdev-all?

I imagine the situation is like:

* -- 2.6.16-rc1  -- * -- * -- 2.6.16-rc2  -- * - -  (linus)
  \                        \              \
* -- * -- * -- * -- * -- * -- * -- * -- * -- M - -  (git-netdev-all)

Then, if you bisect between -rc2 and -rc1, you will never actually get to the git-netdev-all branch, since there are no such commits inbetween -rc2 and -rc1. Even if you consider this:

* -- 2.6.16-rc1  -- * -- * -- 2.6.16-rc2  -- * - -  (linus)
  \              /         \              \
* -- X -- Y -- Z -- A -- * -- * -- * -- * -- M - -  (git-netdev-all)

git-bisect will consider the X, Y, Z commits (since they are part of the ancestry between -rc and -rc2), but not commits from A on - it can't reach them topologically if it considers only commits between -rc1 and -rc2:

* -- 2.6.16-rc1  -- * -- * -- 2.6.16-rc2
  \              /
   - X -- Y -- Z

Now, perhaps what you meant is that "when -rc2 got merged to netdev-all, things were already broken". In this case, what you want to do is to use the commit M as the bisect bad point. Then, bisect will walk this subgraph:

* -- 2.6.16-rc1  -- * -- * -- 2.6.16-rc2
  \              /         \              \
   - X -- Y -- Z -- A -- * -- * -- * -- * -- M

I agree that this can be kind of confusing; I'm not sure how to avoid this. Perhaps git-bisect should warn if when bisecting between Q and P, there exists a path between HEAD and P avoiding Q...?

Dear diary, on Tue, Feb 14, 2006 at 02:15:12AM CET, I got a letter where Petr Baudis <pasky@suse.cz> said that...

I got this one (and consequently, the following one) wrong - obviously, it should read as

       2.6.16-rc1  -- * -- * -- 2.6.16-rc2
                   /
       X -- Y -- Z

since the "asterisk" commit is already behind -rc1.

Pedagogical excursion:

All those commit intervals are really set differences - if you have commit A and commit B,

	[A,B] = B \cup (ancestry(B) \ ancestry(A))

or if you don't like math, color B and all its ancestors blue in your head, and then color all the A ancestors black. The commits that stay blue are in the [A,B] interval.

Andrew Morton <akpm@osdl.org> writes:

Ah. Jeff merges from Linus and that causes things on Linus tree to appear in his tree. So you saw -rc1 and -rc2 in the output, but neither of them may contain the problematic change, and are not good/bad pair at all. They are probably both good ones.

git log output is chronological and there is no guarantee that the ordering has much to do with the actual ordering of commits, especially when merges are involved. In fact, "Jeff's tree only" suggests to me that 2.6.16-rc2 has not merged those changes, but you thought (arguably rightly so) rc1 and rc2 straddled them.

              -rc1                     -rc2
    ---o---o---o---o---o---o---o---o---o---o---o---o--- Linus
                                            \ 
                                             \ 
       ---o---o---o---*---o---o---o---*---o---o---o---o--- Jeff
                       <- ipw2200 ->

So you would want to perhaps pick two commits like the above * and bisect. If the one marked as bad on the Linus tree initially (-rc2) is not bad and does not reach the allegedly bad commit on Jeff's line, there is no way for bisect to find it.

If you are suspecting ipw2200, 2f633db and 747af1e might be a pair of good anchor points to start bisecting.

The way I came up with these two; I should be using gitk for
this kind of thing, but I do not work in X during daytime, so I
am guessing these from:
 
        $ git rev-list --pretty=oneline linus..garzik/netdev |
          grep -C4 -i ipw2200 | less

This gets the list of commits that are on Jeff's tree but not in Linus' in reverse chrono order, and grabs ones with ipw2200 in their titles. It shows that 2f633db is (close to) the latest that touches ipw2200, and 747af1e is (close to) the reasonably old that touches ipw2200. As a review of these two points, I did this:

	$ git log 747af1e..2f633db

Hope it helps this time...