# git performance after directory copy 8 messages from 2010-09-20 to 2010-09-20. Participants: Gaer, A., Michael J Gruber, Matthieu Moy, Johannes Sixt. Thread: https://gitlist.dev/t/25164 ## Gaer, A., 2010-09-20 09:20 Subject: git performance after directory copy Message-ID: URL: https://gitlist.dev/e/C5A8FDEFF7647F4C9CB927D7DEB307730F33B7C9%40ahr075s.basler.corp ``` Hello all, while moving a project directory around I stumbled over an interesting phenomenon. On a copied source directory "git status" seems to be about 3 times slower than on the original directory. Only after a "git reset" both copies behave the same again. Is this connected to the timestamps of files & directories? Actually I would like to move the project directories of several software developers to a new partition and forcing them all to "git reset" in all of their repos is a little bit annoying. Any suggestions how to "repair" the repos less intrusive? Here's how I measured. The trees reside on an ext3 FS. I have lots of free RAM, so after the first operation all further "git status" seem to run from FS cache in RAM. $ git clone $ cd kernel $ time git status # several times! ... $ time git status # On branch master nothing to commit (working directory clean) real 0m0.691s user 0m0.256s sys 0m0.356s $ cd .. $ rsync -a kernel/ kernel2/ $ cd kernel2 $ time git status # several times! ... $ time git status # On branch master nothing to commit (working directory clean) real 0m2.705s user 0m1.724s sys 0m0.816s $ git reset $ time git status # On branch master nothing to commit (working directory clean) real 0m0.704s user 0m0.296s sys 0m0.348s Regards, Andreas. ``` ## Michael J Gruber, 2010-09-20 09:45 Subject: Re: git performance after directory copy Message-ID: <4C972D49.9040306@drmicha.warpmail.net> URL: https://gitlist.dev/e/4C972D49.9040306%40drmicha.warpmail.net In-Reply-To: ``` Gaer, A. venit, vidit, dixit 20.09.2010 11:20: > Hello all, > > while moving a project directory around I stumbled over an interesting > phenomenon. On a copied source directory "git status" seems to be about > 3 times slower than on the original directory. Only after a "git reset" > both copies behave the same again. Is this connected to the timestamps > of files & directories? Actually I would like to move the project > directories of several software developers to a new partition and > forcing them all to "git reset" in all of their repos is a little bit > annoying. Any suggestions how to "repair" the repos less intrusive? > Since you clone from A to B, then copy from B several times, why don't you clone from A several times instead? If it's really about moving across file-system boundaries, then I don't think there's a way around: you need to refresh the index with the changed inodes information. Michael ``` ## Matthieu Moy, 2010-09-20 09:56 Subject: Re: git performance after directory copy Message-ID: URL: https://gitlist.dev/e/vpq62y0hh63.fsf%40bauges.imag.fr In-Reply-To: ``` "Gaer, A." writes: > On a copied source directory "git status" seems to be about > 3 times slower than on the original directory. It is expected that the first "git status" be slower. It will most likely have to actually re-diff the files and update the index stat-cache. But I'm surprised that the next "git status" are still slow. Other people may get a better explanation, but this very much looks like a bug. Are you sure you don't have any permission problem, like a read-only .git/index? To investigate a bit, you can play with diff.autorefreshindex. When set to false, "git diff" will tell you about the files which are identical, but do not have the same stat information (and hence, that Git has to re-diff). For example: $ git config diff.autorefreshindex false $ git diff # no output: bar.txt exists, but is up to date. $ touch bar.txt $ git diff diff --git a/bar.txt b/bar.txt # No actual diff, but the file appears since its stat information is # different. $ git diff diff --git a/bar.txt b/bar.txt # Further "git diff" behave the same. $ git status ... $ git diff # git status did update the stat-cache, hence, no output from git diff # anymore. Another diagnosis tool would be "strace -e open git status,lstat64". For example: $ touch bar.txt $ strace -e open,lstat64 git status |& grep bar.txt lstat64("bar.txt", {st_mode=S_IFREG|0644, st_size=30, ...}) = 0 open("bar.txt", O_RDONLY|O_LARGEFILE) = 3 $ strace -e open,lstat64 git status |& grep bar.txt lstat64("bar.txt", {st_mode=S_IFREG|0644, st_size=30, ...}) = 0 -- Matthieu Moy http://www-verimag.imag.fr/~moy/ ``` ## Michael J Gruber, 2010-09-20 10:54 Subject: Re: git performance after directory copy Message-ID: <4C973D5F.4080807@drmicha.warpmail.net> URL: https://gitlist.dev/e/4C973D5F.4080807%40drmicha.warpmail.net In-Reply-To: ``` Matthieu Moy venit, vidit, dixit 20.09.2010 11:56: > "Gaer, A." writes: > >> On a copied source directory "git status" seems to be about >> 3 times slower than on the original directory. > > It is expected that the first "git status" be slower. It will most > likely have to actually re-diff the files and update the index > stat-cache. > > But I'm surprised that the next "git status" are still slow. Other > people may get a better explanation, but this very much looks like a > bug. In the OP's case, the inode/dev info differs (after copying, or after moving across FS boundaries). I don't think "git status" updates these. I noticed (and reported) something like this a while ago when, after a reboot, I would get warnings about crossing FS boundaries. Only a "rm .git/index && git reset --hard" was able to refresh the index properly. This case seems to be similar, although a simple "git reset --hard" appears to suffice. Michael ``` ## Gaer, A., 2010-09-20 11:37 Subject: AW: git performance after directory copy Message-ID: URL: https://gitlist.dev/e/C5A8FDEFF7647F4C9CB927D7DEB307730F33B85B%40ahr075s.basler.corp In-Reply-To: ``` Hello all, thanks Matthieu for your tip. "git diff" without "autorefreshindex" shows that all files have different stat info. Calling "git update-index --refresh" seems to be the right cure. What I find a little bit confusing is that "git status" does not warn about such a situation, or even "repair" it. As I said, I wanted to move the project directories of several developers to a new partition without too much interference and the first try didn't succeed because people complained about "bad" "git status" performance (people get used to git rocket-fast performance very soon ;-). Maybe this was introduced in the 1.7 release: "git status" is not "git commit --dry-run" anymore. "git commit --dry-run" does behave as you expect: first call takes a little bit longer, subsequent calls are fast again. BTW, I tested on a system with git version 1.7.1 installed, but release notes do not suggest any changes in that respect in 1.7.2 or 1.7.3. Regards, Andreas. ``` ## Johannes Sixt, 2010-09-20 11:48 Subject: Re: git performance after directory copy Message-ID: <4C974A00.8030905@viscovery.net> URL: https://gitlist.dev/e/4C974A00.8030905%40viscovery.net In-Reply-To: ``` Am 9/20/2010 11:56, schrieb Matthieu Moy: > But I'm surprised that the next "git status" are still slow. Other > people may get a better explanation, but this very much looks like a > bug. Most likely, Andreas works with 1.7.1. From the release notes of 1.7.1.1: * "git status" stopped refreshing the index by mistake in 1.7.1. -- Hannes ``` ## Gaer, A., 2010-09-20 11:53 Subject: AW: git performance after directory copy Message-ID: URL: https://gitlist.dev/e/C5A8FDEFF7647F4C9CB927D7DEB307730F33B865%40ahr075s.basler.corp In-Reply-To: <4C974A00.8030905@viscovery.net> ``` Hello Johannes, you are right. Sorry. I totally overlooked release notes for 1.7.1.1. Thanks for all your help! Regards, Andreas. -----Ursprüngliche Nachricht----- Von: Johannes Sixt [mailto:j.sixt@viscovery.net] Gesendet: Montag, 20. September 2010 13:48 An: Matthieu Moy Cc: Gaer, A.; git@vger.kernel.org Betreff: Re: git performance after directory copy Am 9/20/2010 11:56, schrieb Matthieu Moy: > But I'm surprised that the next "git status" are still slow. Other > people may get a better explanation, but this very much looks like a > bug. Most likely, Andreas works with 1.7.1. From the release notes of 1.7.1.1: * "git status" stopped refreshing the index by mistake in 1.7.1. -- Hannes ``` ## Matthieu Moy, 2010-09-20 13:57 Subject: Re: git performance after directory copy Message-ID: URL: https://gitlist.dev/e/vpqeico8qm1.fsf%40bauges.imag.fr In-Reply-To: <4C974A00.8030905@viscovery.net> ``` Johannes Sixt writes: > Am 9/20/2010 11:56, schrieb Matthieu Moy: >> But I'm surprised that the next "git status" are still slow. Other >> people may get a better explanation, but this very much looks like a >> bug. > > Most likely, Andreas works with 1.7.1. From the release notes of 1.7.1.1: > > * "git status" stopped refreshing the index by mistake in 1.7.1. Nice catch, and this explains why I couldn't reproduce with latest Git. More precisely, it was fixed here: b2f6fd9 t7508: add a test for "git status" in a read-only repository 4bb6644 git status: refresh the index if possible 4c926b3 t7508: add test for "git status" refreshing the index -- Matthieu Moy http://www-verimag.imag.fr/~moy/ ```