git commit results in many lstat()s

Hello,

My high level problem is to speed up git commit on a large repository stored on NFS filesystem. I see via strace that it is slow because it makes a large number (~50,000) of lstat() calls in serial. Every call is a round-trip to the NFS server.

I do not understand why git commit must call lstat() on every file in the repository, even when I specify the name of the file I want to commit on the command line. Can somebody explain why it must call lstat on every file?

My command-line looks like this: git commit -uno -o -m asdf file-to-commit.txt

Secondly, are there any optimizations I can make to avoid this behavior?

Thanks, Matt

"Gumbel, Matthew K" <matthew.k.gumbel@intel.com> writes:

Assuming the "COPYING" and "README.md" files are already tracked:

    $ >COPYING
    $ >README.md
    $ git commit COPYING

would open an editor, in which you would see list of files under "Changes to be committed", "Changes not staged for commit", etc. Among the second class you would see README.md listed.

To figure out what paths are "changed", without having to open all files and compare their contents with what is recorded in the commit you are building on top of, we do lstat(2) to see if the timestamp (and other information in the inode) of the files are the same since you checked them out of HEAD.

    $ git commit --no-status COPYING

would reduce the number of lstat(2) somewhat, because the codepath is told that it does not have to make the list to be shown in the editor. So would

    $ git commit -m "empty COPYING" COPYING

These two only halve the number of lstat(2), by taking advantage of the fact that the list of "modified files" does not have to be built. There probably are other things that can be optimized.

"Junio C Hamano" <jch2355@gmail.com> writes:

Yes, I think that when the user passes --only flag to git-commit, then git does not need to call refresh_cache() in prepare_index() in builtin/commit.c.

I may experiment with that. Do you see any downside or negative side-effects?

Matt

"Gumbel, Matthew K" <matthew.k.gumbel@intel.com> writes:

There may be other fallouts, but one that immediately comes to mind is that it may break pre-commit hook.

When we get "--only", we prepare an temporary index to create the commit out of, and give it to the pre-commit hook. The hook expects that the cached stat information is up-to-date, iow, it does not have to do 'update-index --refresh' before using plumbing commands like "diff-index" to do its own inspection of the working tree.

"Junio C Hamano <mailto:jch2355@gmail.com> writes:

"Gumbel, Matthew K" <matthew.k.gumbel@intel.com> writes:

If pre-commit hook exists, we can fall-back to original behavior and call refresh_cache(). Many repos will not have pre-commit hook and can benefit from the speedup.

I'm testing such a change locally. Git test suite seems to be running for quite a while. Do you know any way to run it in parallel or otherwise speed it up?

Thanks, Matt

On Thu, Feb 02, 2017 at 12:14:30AM +0000, Gumbel, Matthew K wrote:

I usually do something like the following:

  make -j3 all && (cd t && GIT_PROVE_OPTS=-j3 make prove)

This, of course, requires that you have Perl's prove installed, which has been part of core Perl since 5.10.1.