{"thread":{"id":"10852","subject":"Strange \"beagle\" interaction..","startedAt":"2007-11-13T20:56:19Z","lastAt":"2007-11-13T21:58:55Z","messageCount":9,"participants":["Linus Torvalds","J. Bruce Fields","Jon Smirl","David Brown","Mike Hommey"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"59727","messageId":"alpine.LFD.0.9999.0711131241050.2786@woody.linux-foundation.org","threadId":"10852","inReplyTo":null,"subject":"Strange \"beagle\" interaction..","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-11-13T20:56:19Z","receivedAt":"2007-11-13T20:56:19Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nOk, I've made a bugzilla entry for this for the Fedora people, but I \nthought I'd mention something I noticed yesterday but only tracked down \ntoday: it seems like the beagle file indexing code is able to screw up git \nin subtle ways.\n\nI do not know exactly what happens, but the symptoms are random (and \nquite hard-to-trigger) dirty index contents where git believes that some \nset of files are not clean in the index.\n\nI *suspect* that beagle is playing games with the file access times, \ncausing the ctime on disk to not match the ce_ctime in the index file. But \nthat's just a guess.\n\nI'm posting here in case somebody on the list knows what beagle does, or \nsomebody has been bitten by strange behaviour and realizes that he has \nbeagle running and prefers to fix the problem by just disabling beagle \n(which will also be a great boon for performance - beagle seems to be very \ngood at flushing your file caches, but I guess that's not a bug, but a \n\"feature\").\n\nThe easiest way I have found so far to trigger this is to run\n\n\twhile ./t7003-filter-branch.sh -i; do echo ok; done\n\nin the git t/ directory, while at the same time telling beagle to index \njust that git/t/ directory. That seems to trigger a failure on subtest 17 \nfairly reliably (not the first time through the loop, but *eventually* - \nit takes a few minutes). I think it's because \"git filter-branch\" requires \nthe index to be clean.\n\n(But I've also seen it fail on subtest 4).\n\nI opened bugzilla\n\n\thttps://bugzilla.redhat.com/show_bug.cgi?id=380791\n\nfor this, since I consider it a beagle bug (indexing shouldn't change \ndirectory state, and if beagle wants to avoid changing access times, it \nshould use O_NOATIME). But I don't actually know exactly what it is that \ncauses problems, so if somebody is interested and tries to figure this \nout, that would probably be good.\n\n\t\t\tLinus\n"},{"id":"59728","messageId":"20071113210354.GD22590@fieldses.org","threadId":"10852","inReplyTo":"alpine.LFD.0.9999.0711131241050.2786@woody.linux-foundation.org","subject":"Re: Strange \"beagle\" interaction..","fromName":"J. Bruce Fields","fromEmail":"bfields@fieldses.org","sentAt":"2007-11-13T21:03:54Z","receivedAt":"2007-11-13T21:03:54Z","isPatch":false,"sender":{"key":"bfields@citi.umich.edu","avatar":null},"body":"On Tue, Nov 13, 2007 at 12:56:19PM -0800, Linus Torvalds wrote:\n> \n> Ok, I've made a bugzilla entry for this for the Fedora people, but I \n> thought I'd mention something I noticed yesterday but only tracked down \n> today: it seems like the beagle file indexing code is able to screw up git \n> in subtle ways.\n> \n> I do not know exactly what happens, but the symptoms are random (and \n> quite hard-to-trigger) dirty index contents where git believes that some \n> set of files are not clean in the index.\n> \n> I *suspect* that beagle is playing games with the file access times, \n> causing the ctime on disk to not match the ce_ctime in the index file. But \n> that's just a guess.\n> \n> I'm posting here in case somebody on the list knows what beagle does, or \n> somebody has been bitten by strange behaviour and realizes that he has \n> beagle running and prefers to fix the problem by just disabling beagle \n> (which will also be a great boon for performance - beagle seems to be very \n> good at flushing your file caches, but I guess that's not a bug, but a \n> \"feature\").\n\nLast I ran across this, I believe I found it was adding extended\nattributes to the file. I think it's something like\n\n\tgetfattr -d\n\nto show all the extended attributes set on the file. Does that show\nanything?\n\nYeah, I just turned off beagle. It looked to me like it was doing\nsomething wrongheaded.\n\n--b.\n\n> \n> The easiest way I have found so far to trigger this is to run\n> \n> \twhile ./t7003-filter-branch.sh -i; do echo ok; done\n> \n> in the git t/ directory, while at the same time telling beagle to index \n> just that git/t/ directory. That seems to trigger a failure on subtest 17 \n> fairly reliably (not the first time through the loop, but *eventually* - \n> it takes a few minutes). I think it's because \"git filter-branch\" requires \n> the index to be clean.\n> \n> (But I've also seen it fail on subtest 4).\n> \n> I opened bugzilla\n> \n> \thttps://bugzilla.redhat.com/show_bug.cgi?id=380791\n> \n> for this, since I consider it a beagle bug (indexing shouldn't change \n> directory state, and if beagle wants to avoid changing access times, it \n> should use O_NOATIME). But I don't actually know exactly what it is that \n> causes problems, so if somebody is interested and tries to figure this \n> out, that would probably be good.\n> \n> \t\t\tLinus\n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"59733","messageId":"20071113212121.GE22590@fieldses.org","threadId":"10852","inReplyTo":"20071113210354.GD22590@fieldses.org","subject":"Re: Strange \"beagle\" interaction..","fromName":"J. Bruce Fields","fromEmail":"bfields@fieldses.org","sentAt":"2007-11-13T21:21:21Z","receivedAt":"2007-11-13T21:21:21Z","isPatch":false,"sender":{"key":"bfields@citi.umich.edu","avatar":null},"body":"On Tue, Nov 13, 2007 at 04:03:54PM -0500, J. Bruce Fields wrote:\n> On Tue, Nov 13, 2007 at 12:56:19PM -0800, Linus Torvalds wrote:\n> > \n> > Ok, I've made a bugzilla entry for this for the Fedora people, but I \n> > thought I'd mention something I noticed yesterday but only tracked down \n> > today: it seems like the beagle file indexing code is able to screw up git \n> > in subtle ways.\n> > \n> > I do not know exactly what happens, but the symptoms are random (and \n> > quite hard-to-trigger) dirty index contents where git believes that some \n> > set of files are not clean in the index.\n> > \n> > I *suspect* that beagle is playing games with the file access times, \n> > causing the ctime on disk to not match the ce_ctime in the index file. But \n> > that's just a guess.\n> > \n> > I'm posting here in case somebody on the list knows what beagle does, or \n> > somebody has been bitten by strange behaviour and realizes that he has \n> > beagle running and prefers to fix the problem by just disabling beagle \n> > (which will also be a great boon for performance - beagle seems to be very \n> > good at flushing your file caches, but I guess that's not a bug, but a \n> > \"feature\").\n> \n> Last I ran across this, I believe I found it was adding extended\n> attributes to the file. I think it's something like\n> \n> \tgetfattr -d\n> \n> to show all the extended attributes set on the file. Does that show\n> anything?\n\nBy the way, on Debian or Ubuntu, at least, that requires an \"apt-get\ninstall attr\" first.\n\n> \n> Yeah, I just turned off beagle. It looked to me like it was doing\n> something wrongheaded.\n\nJust looking at the attribute names and taking a wild guess, it looked\nto me like beagle was computing a checksum of each file's data and\ncomparing it to a checksum previously stored in an xattr, and using that\nto decide whether to reindex the file data.\n\nWith the result that to check whether anything's changed when it starts\nup again it has to read through the entire filesystem's data.\n\nMaybe I'm wrong--I hope so. I'd love to know.\n\n--b.\n"},{"id":"59736","messageId":"alpine.LFD.0.9999.0711131326310.2786@woody.linux-foundation.org","threadId":"10852","inReplyTo":"20071113210354.GD22590@fieldses.org","subject":"Re: Strange \"beagle\" interaction..","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-11-13T21:30:45Z","receivedAt":"2007-11-13T21:30:45Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 13 Nov 2007, J. Bruce Fields wrote:\n> \n> Last I ran across this, I believe I found it was adding extended\n> attributes to the file.\n\nYeah, I just straced it and found the same thing. It's saving fingerprints \nand mtimes to files in the extended attributes.\n\n> Yeah, I just turned off beagle. It looked to me like it was doing\n> something wrongheaded.\n\nGaah. The problem is, setting xattrs does actually change ctime. Which \nmeans that if we want to make git play nice with beagle, I guess we have \nto just remove the comparison of ctime.\n\nOh, well. Git doesn't *require* it, but I like the notion of checking the \ninode really really carefully. But it looks like it may not be an option, \nbecause of file indexers hiding stuff behind our backs.\n\nOr we could just tell people not to run beagle on their git trees, but I \nsuspect some people will actually *want* to. Even if it flushes their disk \ncaches.\n\n\t\tLinus\n"},{"id":"59743","messageId":"9e4733910711131344t381b939dg47b5e078c52be3b2@mail.gmail.com","threadId":"10852","inReplyTo":"alpine.LFD.0.9999.0711131326310.2786@woody.linux-foundation.org","subject":"Re: Strange \"beagle\" interaction..","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-11-13T21:44:33Z","receivedAt":"2007-11-13T21:44:33Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 11/13/07, Linus Torvalds wrote:\n>\n>\n> On Tue, 13 Nov 2007, J. Bruce Fields wrote:\n> >\n> > Last I ran across this, I believe I found it was adding extended\n> > attributes to the file.\n>\n> Yeah, I just straced it and found the same thing. It's saving fingerprints\n> and mtimes to files in the extended attributes.\n\nThings like Beagle need a guaranteed log of global inotify events.\nThat would let them efficiently find changes made since the last time\nthey updated their index.\n\nRight now every time Beagle starts it hasn't got a clue what has\nchanged in the file system since it was last run. This forces Beagle\nto rescan the entire filesystem every time it is started. The xattrs\nare used as cache to reduce this load somewhat.\n\nA better solution would be for the kernel to log inotify events to\ndisk in a manner that survives reboots. When Beagle starts it would\nlocate its last checkpoint and then process the logged inotify events\nfrom that time forward. This inotify logging needs to be bullet proof\nor it will mess up your Beagle index.\n\nLogged files systems already contain the logged inotify data (in their\nown internal form). There's just no universal API for retrieving it in\na file system independent manner.\n\n>\n> > Yeah, I just turned off beagle. It looked to me like it was doing\n> > something wrongheaded.\n>\n> Gaah. The problem is, setting xattrs does actually change ctime. Which\n> means that if we want to make git play nice with beagle, I guess we have\n> to just remove the comparison of ctime.\n>\n> Oh, well. Git doesn't *require* it, but I like the notion of checking the\n> inode really really carefully. But it looks like it may not be an option,\n> because of file indexers hiding stuff behind our backs.\n>\n> Or we could just tell people not to run beagle on their git trees, but I\n> suspect some people will actually *want* to. Even if it flushes their disk\n> caches.\n>\n> Linus\n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"59748","messageId":"20071113214934.GA30950@old.davidb.org","threadId":"10852","inReplyTo":"9e4733910711131344t381b939dg47b5e078c52be3b2@mail.gmail.com","subject":"Re: Strange \"beagle\" interaction..","fromName":"David Brown","fromEmail":"git@davidb.org","sentAt":"2007-11-13T21:49:34Z","receivedAt":"2007-11-13T21:49:34Z","isPatch":false,"sender":{"key":"git@davidb.org","avatar":"https://gravatar.com/avatar/94c86a2938470a74c2eac5e2b69afc0871f79a660295c02219597aba8cb101c1?d=mp&s=160"},"body":"On Tue, Nov 13, 2007 at 04:44:33PM -0500, Jon Smirl wrote:\n\n>A better solution would be for the kernel to log inotify events to\n>disk in a manner that survives reboots. When Beagle starts it would\n>locate its last checkpoint and then process the logged inotify events\n>from that time forward. This inotify logging needs to be bullet proof\n>or it will mess up your Beagle index.\n\nPerhaps something similar to FsEvents on OSX which is a daemon that\ninterfaces with the OS to record this very information.\n\nIt only works across clean reboots, but it does work there. Do a bad\nshutdown, and your next backup or index take a long time to go scan\neverything.\n\nIt would also be wonderful to have this for backups as well.\n\nDave\n"},{"id":"59750","messageId":"20071113215047.GG22590@fieldses.org","threadId":"10852","inReplyTo":"9e4733910711131344t381b939dg47b5e078c52be3b2@mail.gmail.com","subject":"Re: Strange \"beagle\" interaction..","fromName":"J. Bruce Fields","fromEmail":"bfields@fieldses.org","sentAt":"2007-11-13T21:50:47Z","receivedAt":"2007-11-13T21:50:47Z","isPatch":false,"sender":{"key":"bfields@citi.umich.edu","avatar":null},"body":"On Tue, Nov 13, 2007 at 04:44:33PM -0500, Jon Smirl wrote:\n> On 11/13/07, Linus Torvalds wrote:\n> >\n> >\n> > On Tue, 13 Nov 2007, J. Bruce Fields wrote:\n> > >\n> > > Last I ran across this, I believe I found it was adding extended\n> > > attributes to the file.\n> >\n> > Yeah, I just straced it and found the same thing. It's saving fingerprints\n> > and mtimes to files in the extended attributes.\n> \n> Things like Beagle need a guaranteed log of global inotify events.\n> That would let them efficiently find changes made since the last time\n> they updated their index.\n\nWouldn't a simple change-attribute get you most of the way there? All\nyou need is a number that's guaranteed to increase any time a file is\nupdated.\n\nLacking that, git's current approach (snapshot all the stat data, then\nlook closer at any files that appear to have been touched within a\nsecond of the stat) seems pretty sensible.\n\n--b.\n\n> Right now every time Beagle starts it hasn't got a clue what has\n> changed in the file system since it was last run. This forces Beagle\n> to rescan the entire filesystem every time it is started. The xattrs\n> are used as cache to reduce this load somewhat.\n> \n> A better solution would be for the kernel to log inotify events to\n> disk in a manner that survives reboots. When Beagle starts it would\n> locate its last checkpoint and then process the logged inotify events\n> from that time forward. This inotify logging needs to be bullet proof\n> or it will mess up your Beagle index.\n> \n> Logged files systems already contain the logged inotify data (in their\n> own internal form). There's just no universal API for retrieving it in\n> a file system independent manner.\n"},{"id":"59754","messageId":"20071113215528.GA30695@glandium.org","threadId":"10852","inReplyTo":"alpine.LFD.0.9999.0711131241050.2786@woody.linux-foundation.org","subject":"Re: Strange \"beagle\" interaction..","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2007-11-13T21:55:28Z","receivedAt":"2007-11-13T21:55:28Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Nov 13, 2007 at 12:56:19PM -0800, Linus Torvalds wrote:\n> \n> Ok, I've made a bugzilla entry for this for the Fedora people, but I \n> thought I'd mention something I noticed yesterday but only tracked down \n> today: it seems like the beagle file indexing code is able to screw up git \n> in subtle ways.\n> \n> I do not know exactly what happens, but the symptoms are random (and \n> quite hard-to-trigger) dirty index contents where git believes that some \n> set of files are not clean in the index.\n> \n> I *suspect* that beagle is playing games with the file access times, \n> causing the ctime on disk to not match the ce_ctime in the index file. But \n> that's just a guess.\n(...)\n\nIIRC, beagle stores a bunch of useful information for itself in extended\nattributes on indexed files. It is likely that it's that that is\ntampering with the file stats.\n\nMike\n"},{"id":"59755","messageId":"9e4733910711131358w4a9c5f73yb0411691a3d91beb@mail.gmail.com","threadId":"10852","inReplyTo":"20071113215047.GG22590@fieldses.org","subject":"Re: Strange \"beagle\" interaction..","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-11-13T21:58:55Z","receivedAt":"2007-11-13T21:58:55Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 11/13/07, J. Bruce Fields wrote:\n> On Tue, Nov 13, 2007 at 04:44:33PM -0500, Jon Smirl wrote:\n> > On 11/13/07, Linus Torvalds wrote:\n> > >\n> > >\n> > > On Tue, 13 Nov 2007, J. Bruce Fields wrote:\n> > > >\n> > > > Last I ran across this, I believe I found it was adding extended\n> > > > attributes to the file.\n> > >\n> > > Yeah, I just straced it and found the same thing. It's saving fingerprints\n> > > and mtimes to files in the extended attributes.\n> >\n> > Things like Beagle need a guaranteed log of global inotify events.\n> > That would let them efficiently find changes made since the last time\n> > they updated their index.\n>\n> Wouldn't a simple change-attribute get you most of the way there? All\n> you need is a number that's guaranteed to increase any time a file is\n> updated.\n\nYou still need to look at every file in the file system. People can\nhave many millions of files in their file systems (I have two million\nin mine and that's small). The inotify log is the most efficient\nsolution.\n\nI've turned Beagle off simply because it beats on my disk for an hour\nafter I reboot.\n\n>\n> Lacking that, git's current approach (snapshot all the stat data, then\n> look closer at any files that appear to have been touched within a\n> second of the stat) seems pretty sensible.\n>\n> --b.\n>\n> > Right now every time Beagle starts it hasn't got a clue what has\n> > changed in the file system since it was last run. This forces Beagle\n> > to rescan the entire filesystem every time it is started. The xattrs\n> > are used as cache to reduce this load somewhat.\n> >\n> > A better solution would be for the kernel to log inotify events to\n> > disk in a manner that survives reboots. When Beagle starts it would\n> > locate its last checkpoint and then process the logged inotify events\n> > from that time forward. This inotify logging needs to be bullet proof\n> > or it will mess up your Beagle index.\n> >\n> > Logged files systems already contain the logged inotify data (in their\n> > own internal form). There's just no universal API for retrieving it in\n> > a file system independent manner.\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"}]}