{"thread":{"id":"2390","subject":"Real-life kernel debugging scenario","startedAt":"2005-11-08T00:51:50Z","lastAt":"2005-11-09T18:36:50Z","messageCount":7,"participants":["walt","David Lang","Junio C Hamano","Linus Torvalds","wa1ter@myrealbox.com","Jon Loeliger"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"11288","messageId":"dkosr7$f4s$1@sea.gmane.org","threadId":"2390","inReplyTo":null,"subject":"Real-life kernel debugging scenario","fromName":"walt","fromEmail":"wa1ter@myrealbox.com","sentAt":"2005-11-08T00:51:50Z","receivedAt":"2005-11-08T00:51:50Z","isPatch":false,"sender":{"key":"wa1ter@myrealbox.com","avatar":null},"body":"This describes a real problem I've had twice in the last two\nyears while tracking Linus's kernel tree:\n\nI update my local kernel sources every morning using cg-update\n(formerly bk-pull) and compile and install and reboot the new\nkernel.\n\nOkay. On rare occasions I get a kernel panic on reboot. So...\nI know that something Linus committed in the last 24 hours is\nresponsible for the problem.\n\nThe last two times this happened I was able to guess which\ncommit caused the problem and I emailed the developer off-\nlist and got the problem fixed very quickly. (This is why\nI love open-source software!)\n\nMy worry: what happens when I'm not smart enough to guess\nwhich developer to email? My first instinct is to back out\nthe most recent commits one-by-one until the bug goes away.\n\nFirst: is this an optimal tactic?\n\nSecond: how to back out individual commits using git or\ncogito? I suppose this is already spelled out in the docs,\nbut I invite everyone to point me to the relevant places\nin the docs that have escaped my attention so far.\n\nThanks!\n"},{"id":"11290","messageId":"Pine.LNX.4.62.0511071657030.9339@qynat.qvtvafvgr.pbz","threadId":"2390","inReplyTo":"dkosr7$f4s$1@sea.gmane.org","subject":"Re: Real-life kernel debugging scenario","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2005-11-08T00:59:58Z","receivedAt":"2005-11-08T00:59:58Z","isPatch":false,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Mon, 7 Nov 2005, walt wrote:\n\n> My worry: what happens when I'm not smart enough to guess\n> which developer to email? My first instinct is to back out\n> the most recent commits one-by-one until the bug goes away.\n>\n> First: is this an optimal tactic?\n\nit will work, but it's not optimal. this is exactly what git bisect is \ndesigned for.\n\nyou tell it that the prior night's version was good and the current \nversion is bad. It picks a version 'halfway' in between the two and you \ntest that. tell git if the test failed or not and it will then give \nyou the next one to try. repeat until you identify exactly which commit \ntriggers the problem.\n\nunfortunantly I can't trivially point you at the right place in the docs.\n\nDavid Lang\n\n-- \nThere are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.\n -- C.A.R. Hoare\n"},{"id":"11291","messageId":"7vwtjkncyd.fsf@assigned-by-dhcp.cox.net","threadId":"2390","inReplyTo":"dkosr7$f4s$1@sea.gmane.org","subject":"Re: Real-life kernel debugging scenario","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-08T01:30:18Z","receivedAt":"2005-11-08T01:30:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"walt writes:\n\n> First: is this an optimal tactic?\n\nNot if you are a git user.\n\n> Second: how to back out individual commits using git or\n> cogito? I suppose this is already spelled out in the docs,\n> but I invite everyone to point me to the relevant places\n> in the docs that have escaped my attention so far.\n\ngit-bisect(1).\n"},{"id":"11292","messageId":"Pine.LNX.4.64.0511071721200.3247@g5.osdl.org","threadId":"2390","inReplyTo":"dkosr7$f4s$1@sea.gmane.org","subject":"Re: Real-life kernel debugging scenario","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-08T01:31:34Z","receivedAt":"2005-11-08T01:31:34Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 7 Nov 2005, walt wrote:\n> \n> Okay. On rare occasions I get a kernel panic on reboot. So...\n> I know that something Linus committed in the last 24 hours is\n> responsible for the problem.\n> \n> The last two times this happened I was able to guess which\n> commit caused the problem and I emailed the developer off-\n> list and got the problem fixed very quickly. (This is why\n> I love open-source software!)\n> \n> My worry: what happens when I'm not smart enough to guess\n> which developer to email? My first instinct is to back out\n> the most recent commits one-by-one until the bug goes away.\n\nNo. This is what \"git bisect\" is there for.\n\nIt works wonderfully well. To the point where I don't even try to be smart \nabout things: I've had two cases in the last month of merges that caused \nproblems for me, and instead of even trying to guess which patch it was, I \njust bisected it.\n\n> First: is this an optimal tactic?\n> \n> Second: how to back out individual commits using git or\n> cogito? I suppose this is already spelled out in the docs,\n> but I invite everyone to point me to the relevant places\n> in the docs that have escaped my attention so far.\n\nFirst, try \"git bisect\". It doesn't depend on backing out individual \ncommits, instead it starts a special \"bisect\" branch, and (as the name \nimplies) does a binary search within that branch to figure out what is \nwrong.\n\nThe way to use \"git bisect\" couldn't be easier.\n\nFigure out what the oldest bad state you know about is (that's usually the \nhead of \"master\", since that's what you just tried to boot and failed at). \nAlso, figure out the most recent known-good commit (usually the _previous_ \nkernel you ran: and if you've only done a single \"pull\" in between, it \nwill be ORIG_HEAD).\n\nThen do\n\n\tgit bisect start\n\tgit bisect bad master\t\t<- mark \"master\" as the bad state\n\tgit bisect good ORIG_HEAD\t<- mark ORIG_HEAD as good (or\n\t\t\t\t\t whatever other known-good \n\t\t\t\t\t thing you booted laste)\n\nand at this point \"git bisect\" will churn for a while, and tell you what \nthe mid-point between those two commits are, and check that state out as \nthe head of the bew \"bisect\" branch.\n\nCompile and reboot.\n\nIf it's good, just do\n\n\tgit bisect good\t\t<- mark current head as good\n\notherwise, reboot into a good kernel instead, and do (surprise surprise, \ngit really is very intuitive):\n\n\tgit bisect bad\t\t<- mark current head as bad\n\nand whatever you do, git will select a new half-way point. Do this for a \nwhile, until git tells you exactly which commit was the first bad commit. \nThat's your culprit.\n\nIt really works wonderfully well, except for the case where there was \n_another_ commit that broke something in between, like introduced some \nstupid compile error. In that case you should not mark that commit good or \nbad: you should try to find another commit close-by, and do a \"git reset \n--hard \" to try out _that_ commit instead, and then test that \ninstead (and mark it good or bad).\n\nYou can do \"git bisect visualize\" while you do all this to see what's \ngoing on by starting up gitk on the bisection range.\n\nFinally, once you've figured out exactly which commit was bad, you can \nthen go back to the master branch, and try reverting just that commit:\n\n\tgit checkout master\n\tgit revert \n\nto verify that the top-of-kernel works with that single commit reverted.\n\n\t\tLinus\n"},{"id":"11396","messageId":"Pine.LNX.4.64.0511090901370.6744@x2.ybpnyarg","threadId":"2390","inReplyTo":"Pine.LNX.4.64.0511071721200.3247@g5.osdl.org","subject":"Re: Real-life kernel debugging scenario","fromName":"","fromEmail":"wa1ter@myrealbox.com","sentAt":"2005-11-09T17:40:06Z","receivedAt":"2005-11-09T17:40:06Z","isPatch":false,"sender":{"key":"wa1ter@myrealbox.com","avatar":null},"body":"\n\nOn Mon, 7 Nov 2005, Linus Torvalds wrote:\n\n[...]\n> The way to use \"git bisect\" couldn't be easier.\n\nI see that Junio just added your entire response to Documentation/howto/\nisolate-bugs-with-bisect.txt where even I could manage to find it :o)\n\n> Also, figure out the most recent known-good commit (usually the _previous_\n> kernel you ran: and if you've only done a single \"pull\" in between, it\n> will be ORIG_HEAD).\n\nThe built-in variable ORIG_HEAD isn't explained anywhere AFAICT, or at\nleast it wasn't until today. Are there other such magic variables which\nmight be useful for us to know about?\n"},{"id":"11397","messageId":"1131560233.23046.26.camel@cashmere.sps.mot.com","threadId":"2390","inReplyTo":"Pine.LNX.4.64.0511090901370.6744@x2.ybpnyarg","subject":"Re: Real-life kernel debugging scenario","fromName":"Jon Loeliger","fromEmail":"jdl@freescale.com","sentAt":"2005-11-09T18:17:13Z","receivedAt":"2005-11-09T18:17:13Z","isPatch":false,"sender":{"key":"jdl@jdl.com","avatar":"https://gravatar.com/avatar/75ce9a10b151acd2c28ec4ab2136dba7b2ff1634530bd04b155981a749d08a64?d=mp&s=160"},"body":"On Wed, 2005-11-09 at 11:40, wa1ter@myrealbox.com wrote:\n> \n> On Mon, 7 Nov 2005, Linus Torvalds wrote:\n> \n> [...]\n> > The way to use \"git bisect\" couldn't be easier.\n> \n> I see that Junio just added your entire response to\n> Documentation/howto/\n> isolate-bugs-with-bisect.txt where even I could manage to find it :o)\n\nWhy, thank you. :-)\n\n\n> The built-in variable ORIG_HEAD isn't explained anywhere AFAICT, or at\n> least it wasn't until today. Are there other such magic variables\n> which\n> might be useful for us to know about?\n\nHey Junio,\n\nI'm going to call \"I told you so!\" on this one! :-)\n\nRef:\n Message-ID: \n From: Jon Loeliger \n Subject: Now What?\n Date: Wed, 2 Nov 2005 18:30:37 -0700\n\nWhere in I mumbled:\n\n I feel that an explanation of all of the behind-the-scripts-\n in-.git communication files is needed. In particular these:\n\n FETCH_HEAD\n MERGE_HEAD\n LAST_MERGE\n MERGE_MSG\n\n These need to be mentioned and explained because they\n frequently form exactly the critical missing link or\n starting point after a failed fetch or merge.\n\nYou know.\n\njdl\n"},{"id":"11399","messageId":"Pine.LNX.4.64.0511091028241.3247@g5.osdl.org","threadId":"2390","inReplyTo":"Pine.LNX.4.64.0511090901370.6744@x2.ybpnyarg","subject":"Re: Real-life kernel debugging scenario","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-09T18:36:50Z","receivedAt":"2005-11-09T18:36:50Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 9 Nov 2005, wa1ter@myrealbox.com wrote:\n> \n> The built-in variable ORIG_HEAD isn't explained anywhere AFAICT, or at\n> least it wasn't until today. Are there other such magic variables which\n> might be useful for us to know about?\n\nNo, ORIG_HEAD is pretty much it.\n\nThere's a few special files that a non-committed merge leaves around \n(either because you asked it to not be committed, or because it had \nclashes and requires manual fixing), but they are not normally useful to \nany regular people. You can poke around in the \".git\" directory after such \na merge if you want (MERGE_HEAD, MERGE_MSG).\n\nORIG_HEAD is _very_ useful, though, and I use it all the time. Any time \nyou've pulled something from somebody else, and you wonder what you \npulled, just go\n\n\tgitk ORIG_HEAD..\n\nand you'll see exactly what new stuff you got in your branch. I often do \nthat command line several times a day as I merge stuff. Otherwise the new \nstuff tends to be hidden in the noise.\n\n(There's also FETCH_HEAD, which is the head of the last fetch, so if you \naren't interested in any potential merge, you could instead use the range \nORIG_HEAD..FETCH_HEAD, but quite frankly, I doubt anybody really cares \nexcept for the internal git fetching/pulling logic).\n\n\t\tLinus\n"}]}