threads / discuss / 25164

git performance after directory copy

Subject: git performance after directory copy

## tl;dr

8 messages between Sep 20, 2010 and Sep 20, 2010.

replies: 7people: 4as markdown or json

Gaer, A.· Sep 20, 2010, 09:20 UTC · lore
Hello all,

while moving a project directory around I stumbled over an interesting phenomenon. On a copied source directory "git status" seems to be about 3 times slower than on the original directory. Only after a "git reset" both copies behave the same again. Is this connected to the timestamps of files & directories? Actually I would like to move the project directories of several software developers to a new partition and forcing them all to "git reset" in all of their repos is a little bit annoying. Any suggestions how to "repair" the repos less intrusive?

Here's how I measured. The trees reside on an ext3 FS. I have lots of free RAM, so after the first operation all further "git status" seem to run from FS cache in RAM.

$ git clone <path to your preferred kernel> $ cd kernel $ time git status # several times! ... $ time git status # On branch master nothing to commit (working directory clean)

real 0m0.691s user 0m0.256s sys 0m0.356s

$ cd .. $ rsync -a kernel/ kernel2/ $ cd kernel2 $ time git status # several times! ... $ time git status # On branch master nothing to commit (working directory clean)

real 0m2.705s user 0m1.724s sys 0m0.816s

$ git reset $ time git status # On branch master nothing to commit (working directory clean)

real 0m0.704s user 0m0.296s sys 0m0.348s

Regards,
 Andreas.
Michael J Gruber· Sep 20, 2010, 09:45 UTC · re: Gaer, A. · lore

Re: git performance after directory copy

Gaer, A. venit, vidit, dixit 20.09.2010 11:20:
Show 11 quoted lines
> Hello all,
> 
> while moving a project directory around I stumbled over an interesting
> phenomenon. On a copied source directory "git status" seems to be about
> 3 times slower than on the original directory. Only after a "git reset"
> both copies behave the same again. Is this connected to the timestamps
> of files & directories? Actually I would like to move the project
> directories of several software developers to a new partition and
> forcing them all to "git reset" in all of their repos is a little bit
> annoying. Any suggestions how to "repair" the repos less intrusive?
>

Since you clone from A to B, then copy from B several times, why don't you clone from A several times instead?

If it's really about moving across file-system boundaries, then I don't think there's a way around: you need to refresh the index with the changed inodes information.

Michael
Matthieu Moy· Sep 20, 2010, 09:56 UTC · re: Gaer, A. · lore

Re: git performance after directory copy

"Gaer, A." <Andreas.Gaer@baslerweb.com> writes:
> On a copied source directory "git status" seems to be about
> 3 times slower than on the original directory.

It is expected that the first "git status" be slower. It will most likely have to actually re-diff the files and update the index stat-cache.

But I'm surprised that the next "git status" are still slow. Other people may get a better explanation, but this very much looks like a bug.

Are you sure you don't have any permission problem, like a read-only .git/index?

To investigate a bit, you can play with diff.autorefreshindex. When set to false, "git diff" will tell you about the files which are identical, but do not have the same stat information (and hence, that Git has to re-diff). For example:

$ git config diff.autorefreshindex false $ git diff

# no output: bar.txt exists, but is up to date.

$ touch bar.txt $ git diff 

diff --git a/bar.txt b/bar.txt

# No actual diff, but the file appears since its stat information is
# different.

$ git diff
diff --git a/bar.txt b/bar.txt

# Further "git diff" behave the same.

$ git status
...
$ git diff  

# git status did update the stat-cache, hence, no output from git diff
# anymore.

Another diagnosis tool would be "strace -e open git status,lstat64".
For example:

$ touch bar.txt 
$ strace -e open,lstat64 git status |& grep bar.txt
lstat64("bar.txt", {st_mode=S_IFREG|0644, st_size=30, ...}) = 0
open("bar.txt", O_RDONLY|O_LARGEFILE)   = 3
$ strace -e open,lstat64 git status |& grep bar.txt
lstat64("bar.txt", {st_mode=S_IFREG|0644, st_size=30, ...}) = 0

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/
Michael J Gruber· Sep 20, 2010, 10:54 UTC · re: Matthieu Moy · lore

Re: git performance after directory copy

Matthieu Moy venit, vidit, dixit 20.09.2010 11:56:
Show 12 quoted lines
> "Gaer, A." <Andreas.Gaer@baslerweb.com> writes:
> 
>> On a copied source directory "git status" seems to be about
>> 3 times slower than on the original directory.
> 
> It is expected that the first "git status" be slower. It will most
> likely have to actually re-diff the files and update the index
> stat-cache.
> 
> But I'm surprised that the next "git status" are still slow. Other
> people may get a better explanation, but this very much looks like a
> bug.

In the OP's case, the inode/dev info differs (after copying, or after moving across FS boundaries). I don't think "git status" updates these. I noticed (and reported) something like this a while ago when, after a reboot, I would get warnings about crossing FS boundaries. Only a "rm .git/index && git reset --hard" was able to refresh the index properly. This case seems to be similar, although a simple "git reset --hard" appears to suffice.

Michael
Gaer, A.· Sep 20, 2010, 11:37 UTC · re: Matthieu Moy · lore

AW: git performance after directory copy

Hello all,

thanks Matthieu for your tip. "git diff" without "autorefreshindex" shows that all files have different stat info. Calling "git update-index --refresh" seems to be the right cure.

What I find a little bit confusing is that "git status" does not warn about such a situation, or even "repair" it. As I said, I wanted to move the project directories of several developers to a new partition without too much interference and the first try didn't succeed because people complained about "bad" "git status" performance (people get used to git rocket-fast performance very soon ;-).

Maybe this was introduced in the 1.7 release: "git status" is not "git commit --dry-run" anymore. "git commit --dry-run" does behave as you expect: first call takes a little bit longer, subsequent calls are fast again.

BTW, I tested on a system with git version 1.7.1 installed, but release notes do not suggest any changes in that respect in 1.7.2 or 1.7.3.

Regards,
 Andreas.
Johannes Sixt· Sep 20, 2010, 11:48 UTC · re: Matthieu Moy · lore

Re: git performance after directory copy

Am 9/20/2010 11:56, schrieb Matthieu Moy:
> But I'm surprised that the next "git status" are still slow. Other
> people may get a better explanation, but this very much looks like a
> bug.
Most likely, Andreas works with 1.7.1. From the release notes of 1.7.1.1:
 * "git status" stopped refreshing the index by mistake in 1.7.1.
-- Hannes
Gaer, A.· Sep 20, 2010, 11:53 UTC · re: Johannes Sixt · lore

AW: git performance after directory copy

Hello Johannes,
you are right. Sorry. I totally overlooked release notes for 1.7.1.1. Thanks for all your help!
Regards,
 Andreas.
 
-----Ursprüngliche Nachricht-----
Von: Johannes Sixt [mailto:j.sixt@viscovery.net] 
Gesendet: Montag, 20. September 2010 13:48
An: Matthieu Moy
Cc: Gaer, A.; git@vger.kernel.org
Betreff: Re: git performance after directory copy
Am 9/20/2010 11:56, schrieb Matthieu Moy:
> But I'm surprised that the next "git status" are still slow. Other
> people may get a better explanation, but this very much looks like a
> bug.
Most likely, Andreas works with 1.7.1. From the release notes of 1.7.1.1:
 * "git status" stopped refreshing the index by mistake in 1.7.1.
-- Hannes
Matthieu Moy· Sep 20, 2010, 13:57 UTC · re: Johannes Sixt · lore

Re: git performance after directory copy

Johannes Sixt <j.sixt@viscovery.net> writes:
Show 8 quoted lines
> Am 9/20/2010 11:56, schrieb Matthieu Moy:
>> But I'm surprised that the next "git status" are still slow. Other
>> people may get a better explanation, but this very much looks like a
>> bug.
>
> Most likely, Andreas works with 1.7.1. From the release notes of 1.7.1.1:
>
>  * "git status" stopped refreshing the index by mistake in 1.7.1.

Nice catch, and this explains why I couldn't reproduce with latest Git.

More precisely, it was fixed here:

b2f6fd9 t7508: add a test for "git status" in a read-only repository 4bb6644 git status: refresh the index if possible 4c926b3 t7508: add test for "git status" refreshing the index

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

← back to recent threads