{"thread":{"id":"13417","subject":"Sort of a feature proposal","startedAt":"2008-05-07T14:48:17Z","lastAt":"2008-05-07T18:05:20Z","messageCount":8,"participants":["David Kastrup","Nicolas Pitre","Stephen R. van den Berg","Avery Pennarun","Linus Torvalds","Dmitry Potapov"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"76286","messageId":"86fxsutbke.fsf@lola.quinscape.zz","threadId":"13417","inReplyTo":null,"subject":"Sort of a feature proposal","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2008-05-07T14:48:17Z","receivedAt":"2008-05-07T14:48:17Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"\nHi, I have some large git repositories on a USB drive (ext3 file\nsystem). That means that when replugging the drive, the recorded st_dev\ndata in the index is off, meaning that the whole repo directory\nstructure gets reread as the stat data of all directories has changed.\n\nThat's a nuisance. Can't we have some heuristic or configuration option\nwhere we, say, record the st_dev of the _index_ file, and if that has\nchanged, we propagate that change to the st_dev of its contents? I'd\nlike to see something that works more efficiently than rescanning the\nwhole disk every time I hibernate my computer.\n\nThanks,\n\n-- \nDavid Kastrup\n"},{"id":"76296","messageId":"alpine.LFD.1.10.0805071139010.23581@xanadu.home","threadId":"13417","inReplyTo":"86fxsutbke.fsf@lola.quinscape.zz","subject":"Re: Sort of a feature proposal","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-05-07T15:41:12Z","receivedAt":"2008-05-07T15:41:12Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 May 2008, David Kastrup wrote:\n\n> \n> Hi, I have some large git repositories on a USB drive (ext3 file\n> system). That means that when replugging the drive, the recorded st_dev\n> data in the index is off, meaning that the whole repo directory\n> structure gets reread as the stat data of all directories has changed.\n> \n> That's a nuisance. Can't we have some heuristic or configuration option\n> where we, say, record the st_dev of the _index_ file, and if that has\n> changed, we propagate that change to the st_dev of its contents? I'd\n> like to see something that works more efficiently than rescanning the\n> whole disk every time I hibernate my computer.\n\nMaybe simply ignoring st_dev is the solution? I hardly can see what \nvalue it had to the other stat fields.\n\n\nNicolas\n"},{"id":"76297","messageId":"20080507160035.GA16163@cuci.nl","threadId":"13417","inReplyTo":"alpine.LFD.1.10.0805071139010.23581@xanadu.home","subject":"Re: Sort of a feature proposal","fromName":"Stephen R. van den Berg","fromEmail":"srb@cuci.nl","sentAt":"2008-05-07T16:00:35Z","receivedAt":"2008-05-07T16:00:35Z","isPatch":false,"sender":{"key":"srb@cuci.nl","avatar":"https://gravatar.com/avatar/f75389059e827634d38e9df2a9b6ecbd50028b5a454442efa1c7205b7ff29c6a?d=mp&s=160"},"body":"Nicolas Pitre wrote:\n>Maybe simply ignoring st_dev is the solution? I hardly can see what \n>value it had to the other stat fields.\n\nIt determines the scope of st_ino.\n-- \nSincerely, srb@cuci.nl\n Stephen R. van den Berg.\nLady Astor: \"Winston, if you were my husband, I'd put poison in your coffee.\"\n Churchill: \"Nancy, if you were my wife, I'd drink it.\"\n"},{"id":"76298","messageId":"32541b130805070903l195d84bn2ec22e97c81c42df@mail.gmail.com","threadId":"13417","inReplyTo":"alpine.LFD.1.10.0805071139010.23581@xanadu.home","subject":"Re: Sort of a feature proposal","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-05-07T16:03:58Z","receivedAt":"2008-05-07T16:03:58Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 5/7/08, Nicolas Pitre wrote:\n> On Wed, 7 May 2008, David Kastrup wrote:\n> > Hi, I have some large git repositories on a USB drive (ext3 file\n> > system). That means that when replugging the drive, the recorded st_dev\n> > data in the index is off, meaning that the whole repo directory\n> > structure gets reread as the stat data of all directories has changed.\n> >\n> > That's a nuisance. Can't we have some heuristic or configuration option\n> > where we, say, record the st_dev of the _index_ file, and if that has\n> > changed, we propagate that change to the st_dev of its contents? I'd\n> > like to see something that works more efficiently than rescanning the\n> > whole disk every time I hibernate my computer.\n>\n> Maybe simply ignoring st_dev is the solution? I hardly can see what\n> value it had to the other stat fields.\n\nIf I understand correctly, you can be sure a file hasn't changed if it\nhas exactly the same (dev,inode,ctime,length) attributes. If you\ndon't track the dev, you can't be certain whether file attributes look\nidentical but it was actually on another disk, and therefore might\nhave different content after all.\n\nIt's obviously a pretty rare case, but nobody likes a version control\nsystem that works properly \"almost\" all the time :)\n\nHave fun,\n\nAvery\n"},{"id":"76305","messageId":"alpine.LFD.1.10.0805070924300.3024@woody.linux-foundation.org","threadId":"13417","inReplyTo":"86fxsutbke.fsf@lola.quinscape.zz","subject":"Re: Sort of a feature proposal","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-05-07T16:25:55Z","receivedAt":"2008-05-07T16:25:55Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 7 May 2008, David Kastrup wrote:\n> \n> Hi, I have some large git repositories on a USB drive (ext3 file\n> system). That means that when replugging the drive, the recorded st_dev\n> data in the index is off, meaning that the whole repo directory\n> structure gets reread as the stat data of all directories has changed.\n> \n> That's a nuisance. Can't we have some heuristic or configuration option\n> where we, say, record the st_dev of the _index_ file, and if that has\n> changed, we propagate that change to the st_dev of its contents? I'd\n> like to see something that works more efficiently than rescanning the\n> whole disk every time I hibernate my computer.\n\nHmm. We shouldn't even be using st_dev any more.\n\nHow did you compile your git version? By default USE_STDEV should be off, \nand it's been that way for a long time (because st_dev is also not \nreliable on NFS etc).\n\n\t\tLinus\n"},{"id":"76310","messageId":"86k5i6rp23.fsf@lola.quinscape.zz","threadId":"13417","inReplyTo":"alpine.LFD.1.10.0805070924300.3024@woody.linux-foundation.org","subject":"Re: Sort of a feature proposal","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2008-05-07T17:39:48Z","receivedAt":"2008-05-07T17:39:48Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Linus Torvalds writes:\n\n> On Wed, 7 May 2008, David Kastrup wrote:\n>> \n>> Hi, I have some large git repositories on a USB drive (ext3 file\n>> system). That means that when replugging the drive, the recorded st_dev\n>> data in the index is off, meaning that the whole repo directory\n>> structure gets reread as the stat data of all directories has changed.\n>> \n>> That's a nuisance. Can't we have some heuristic or configuration option\n>> where we, say, record the st_dev of the _index_ file, and if that has\n>> changed, we propagate that change to the st_dev of its contents? I'd\n>> like to see something that works more efficiently than rescanning the\n>> whole disk every time I hibernate my computer.\n>\n> Hmm. We shouldn't even be using st_dev any more.\n>\n> How did you compile your git version? By default USE_STDEV should be off, \n> and it's been that way for a long time (because st_dev is also not \n> reliable on NFS etc).\n\nLooks that way in my Makefile. Maybe I am confused: I just did some\ntimings (this is ext3 on a USB drive) and got\n\n git svn rebase\n Current branch master is up to date.\n dak@lisa:/lisa/texlive$ time git svn rebase\n Current branch master is up to date.\n\n real\t0m4.581s\n user\t0m2.244s\n sys\t0m1.492s\n dak@lisa:/lisa/texlive$ cd\n dak@lisa:~$ sudo umount /lisa;sudo mount /dev/mapper/Medion-reps /lisa;cd /lisa/texlive;time git svn rebase\n Current branch master is up to date.\n\n real\t0m53.588s\n user\t0m2.248s\n sys\t0m2.388s\n dak@lisa:/lisa/texlive$ cd;sudo umount /lisa;sudo dmsetup remove /dev/mapper/Medion-reps\n[Unplug and replug the USB drive]\n dak@lisa:~$ sudo mount /dev/mapper/Medion-reps /lisa;cd /lisa/texlive;time git svn rebase\n Current branch master is up to date.\n\n real\t0m53.101s\n user\t0m2.324s\n sys\t0m2.380s\n dak@lisa:/lisa/texlive$ \n\nIf my guess that the device number of LVM does not change when merely\nun- and remounting, but does change when unplugging and replugging is\ncorrect, it would appear that my idea where the time went was wrong and\nthat the device number has nothing whatsoever to do with the large\namount of lookups (this is a USB2.0 device at High Speed).\n\nIs there a way to completely invalidate the disk cache without\nunmounting? How do I verify device numbers?\n\nThanks,\n\n-- \nDavid Kastrup\n"},{"id":"76312","messageId":"37fcd2780805071050n5153f5d2rf31a0f263820e92@mail.gmail.com","threadId":"13417","inReplyTo":"86k5i6rp23.fsf@lola.quinscape.zz","subject":"Re: Sort of a feature proposal","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2008-05-07T17:50:12Z","receivedAt":"2008-05-07T17:50:12Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"> Is there a way to completely invalidate the disk cache without\n> unmounting?\n\necho 3 > /proc/sys/vm/drop_caches\n\nDmitry\n"},{"id":"76313","messageId":"868wymrnvj.fsf@lola.quinscape.zz","threadId":"13417","inReplyTo":"37fcd2780805071050n5153f5d2rf31a0f263820e92@mail.gmail.com","subject":"Re: Sort of a feature proposal","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2008-05-07T18:05:20Z","receivedAt":"2008-05-07T18:05:20Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"\"Dmitry Potapov\" writes:\n\n>> Is there a way to completely invalidate the disk cache without\n>> unmounting?\n>\n> echo 3 > /proc/sys/vm/drop_caches\n\nSigh. It is the disk cache after all. Looks like \"git svn rebase\"\ncan't just work from the index (ok, it checks whether there are\nunstashed modifications). And indeed, flushing the cache will make much\nless of a difference for \"git svn fetch\" than for rebase.\n\nSorry for the noise.\n\n-- \nDavid Kastrup\n"}]}