# Sort of a feature proposal 8 messages from 2008-05-07 to 2008-05-07. Participants: David Kastrup, Nicolas Pitre, Stephen R. van den Berg, Avery Pennarun, Linus Torvalds, Dmitry Potapov. Thread: https://gitlist.dev/t/13417 ## David Kastrup, 2008-05-07 14:48 Subject: Sort of a feature proposal Message-ID: <86fxsutbke.fsf@lola.quinscape.zz> URL: https://gitlist.dev/e/86fxsutbke.fsf%40lola.quinscape.zz ``` Hi, I have some large git repositories on a USB drive (ext3 file system). That means that when replugging the drive, the recorded st_dev data in the index is off, meaning that the whole repo directory structure gets reread as the stat data of all directories has changed. That's a nuisance. Can't we have some heuristic or configuration option where we, say, record the st_dev of the _index_ file, and if that has changed, we propagate that change to the st_dev of its contents? I'd like to see something that works more efficiently than rescanning the whole disk every time I hibernate my computer. Thanks, -- David Kastrup ``` ## Nicolas Pitre, 2008-05-07 15:41 Subject: Re: Sort of a feature proposal Message-ID: URL: https://gitlist.dev/e/alpine.LFD.1.10.0805071139010.23581%40xanadu.home In-Reply-To: <86fxsutbke.fsf@lola.quinscape.zz> ``` On Wed, 7 May 2008, David Kastrup wrote: > > Hi, I have some large git repositories on a USB drive (ext3 file > system). That means that when replugging the drive, the recorded st_dev > data in the index is off, meaning that the whole repo directory > structure gets reread as the stat data of all directories has changed. > > That's a nuisance. Can't we have some heuristic or configuration option > where we, say, record the st_dev of the _index_ file, and if that has > changed, we propagate that change to the st_dev of its contents? I'd > like to see something that works more efficiently than rescanning the > whole disk every time I hibernate my computer. Maybe simply ignoring st_dev is the solution? I hardly can see what value it had to the other stat fields. Nicolas ``` ## Stephen R. van den Berg, 2008-05-07 16:00 Subject: Re: Sort of a feature proposal Message-ID: <20080507160035.GA16163@cuci.nl> URL: https://gitlist.dev/e/20080507160035.GA16163%40cuci.nl In-Reply-To: ``` Nicolas Pitre wrote: >Maybe simply ignoring st_dev is the solution? I hardly can see what >value it had to the other stat fields. It determines the scope of st_ino. -- Sincerely, srb@cuci.nl Stephen R. van den Berg. Lady Astor: "Winston, if you were my husband, I'd put poison in your coffee." Churchill: "Nancy, if you were my wife, I'd drink it." ``` ## Avery Pennarun, 2008-05-07 16:03 Subject: Re: Sort of a feature proposal Message-ID: <32541b130805070903l195d84bn2ec22e97c81c42df@mail.gmail.com> URL: https://gitlist.dev/e/32541b130805070903l195d84bn2ec22e97c81c42df%40mail.gmail.com In-Reply-To: ``` On 5/7/08, Nicolas Pitre wrote: > On Wed, 7 May 2008, David Kastrup wrote: > > Hi, I have some large git repositories on a USB drive (ext3 file > > system). That means that when replugging the drive, the recorded st_dev > > data in the index is off, meaning that the whole repo directory > > structure gets reread as the stat data of all directories has changed. > > > > That's a nuisance. Can't we have some heuristic or configuration option > > where we, say, record the st_dev of the _index_ file, and if that has > > changed, we propagate that change to the st_dev of its contents? I'd > > like to see something that works more efficiently than rescanning the > > whole disk every time I hibernate my computer. > > Maybe simply ignoring st_dev is the solution? I hardly can see what > value it had to the other stat fields. If I understand correctly, you can be sure a file hasn't changed if it has exactly the same (dev,inode,ctime,length) attributes. If you don't track the dev, you can't be certain whether file attributes look identical but it was actually on another disk, and therefore might have different content after all. It's obviously a pretty rare case, but nobody likes a version control system that works properly "almost" all the time :) Have fun, Avery ``` ## Linus Torvalds, 2008-05-07 16:25 Subject: Re: Sort of a feature proposal Message-ID: URL: https://gitlist.dev/e/alpine.LFD.1.10.0805070924300.3024%40woody.linux-foundation.org In-Reply-To: <86fxsutbke.fsf@lola.quinscape.zz> ``` On Wed, 7 May 2008, David Kastrup wrote: > > Hi, I have some large git repositories on a USB drive (ext3 file > system). That means that when replugging the drive, the recorded st_dev > data in the index is off, meaning that the whole repo directory > structure gets reread as the stat data of all directories has changed. > > That's a nuisance. Can't we have some heuristic or configuration option > where we, say, record the st_dev of the _index_ file, and if that has > changed, we propagate that change to the st_dev of its contents? I'd > like to see something that works more efficiently than rescanning the > whole disk every time I hibernate my computer. Hmm. We shouldn't even be using st_dev any more. How did you compile your git version? By default USE_STDEV should be off, and it's been that way for a long time (because st_dev is also not reliable on NFS etc). Linus ``` ## David Kastrup, 2008-05-07 17:39 Subject: Re: Sort of a feature proposal Message-ID: <86k5i6rp23.fsf@lola.quinscape.zz> URL: https://gitlist.dev/e/86k5i6rp23.fsf%40lola.quinscape.zz In-Reply-To: ``` Linus Torvalds writes: > On Wed, 7 May 2008, David Kastrup wrote: >> >> Hi, I have some large git repositories on a USB drive (ext3 file >> system). That means that when replugging the drive, the recorded st_dev >> data in the index is off, meaning that the whole repo directory >> structure gets reread as the stat data of all directories has changed. >> >> That's a nuisance. Can't we have some heuristic or configuration option >> where we, say, record the st_dev of the _index_ file, and if that has >> changed, we propagate that change to the st_dev of its contents? I'd >> like to see something that works more efficiently than rescanning the >> whole disk every time I hibernate my computer. > > Hmm. We shouldn't even be using st_dev any more. > > How did you compile your git version? By default USE_STDEV should be off, > and it's been that way for a long time (because st_dev is also not > reliable on NFS etc). Looks that way in my Makefile. Maybe I am confused: I just did some timings (this is ext3 on a USB drive) and got git svn rebase Current branch master is up to date. dak@lisa:/lisa/texlive$ time git svn rebase Current branch master is up to date. real 0m4.581s user 0m2.244s sys 0m1.492s dak@lisa:/lisa/texlive$ cd dak@lisa:~$ sudo umount /lisa;sudo mount /dev/mapper/Medion-reps /lisa;cd /lisa/texlive;time git svn rebase Current branch master is up to date. real 0m53.588s user 0m2.248s sys 0m2.388s dak@lisa:/lisa/texlive$ cd;sudo umount /lisa;sudo dmsetup remove /dev/mapper/Medion-reps [Unplug and replug the USB drive] dak@lisa:~$ sudo mount /dev/mapper/Medion-reps /lisa;cd /lisa/texlive;time git svn rebase Current branch master is up to date. real 0m53.101s user 0m2.324s sys 0m2.380s dak@lisa:/lisa/texlive$ If my guess that the device number of LVM does not change when merely un- and remounting, but does change when unplugging and replugging is correct, it would appear that my idea where the time went was wrong and that the device number has nothing whatsoever to do with the large amount of lookups (this is a USB2.0 device at High Speed). Is there a way to completely invalidate the disk cache without unmounting? How do I verify device numbers? Thanks, -- David Kastrup ``` ## Dmitry Potapov, 2008-05-07 17:50 Subject: Re: Sort of a feature proposal Message-ID: <37fcd2780805071050n5153f5d2rf31a0f263820e92@mail.gmail.com> URL: https://gitlist.dev/e/37fcd2780805071050n5153f5d2rf31a0f263820e92%40mail.gmail.com In-Reply-To: <86k5i6rp23.fsf@lola.quinscape.zz> ``` > Is there a way to completely invalidate the disk cache without > unmounting? echo 3 > /proc/sys/vm/drop_caches Dmitry ``` ## David Kastrup, 2008-05-07 18:05 Subject: Re: Sort of a feature proposal Message-ID: <868wymrnvj.fsf@lola.quinscape.zz> URL: https://gitlist.dev/e/868wymrnvj.fsf%40lola.quinscape.zz In-Reply-To: <37fcd2780805071050n5153f5d2rf31a0f263820e92@mail.gmail.com> ``` "Dmitry Potapov" writes: >> Is there a way to completely invalidate the disk cache without >> unmounting? > > echo 3 > /proc/sys/vm/drop_caches Sigh. It is the disk cache after all. Looks like "git svn rebase" can't just work from the index (ok, it checks whether there are unstashed modifications). And indeed, flushing the cache will make much less of a difference for "git svn fetch" than for rebase. Sorry for the noise. -- David Kastrup ```