{"thread":{"id":"45014","subject":"git commit results in many lstat()s","startedAt":"2017-02-01T21:45:33Z","lastAt":"2017-02-02T00:26:10Z","messageCount":6,"participants":["Gumbel, Matthew K","Junio C Hamano","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"310664","messageId":"DA0A42D68346B1469147552440A645039A9C56D4@ORSMSX115.amr.corp.intel.com","threadId":"45014","inReplyTo":null,"subject":"git commit results in many lstat()s","fromName":"Gumbel, Matthew K","fromEmail":"matthew.k.gumbel@intel.com","sentAt":"2017-02-01T21:45:25Z","receivedAt":"2017-02-01T21:45:33Z","isPatch":false,"sender":{"key":"matthew.k.gumbel@intel.com","avatar":"https://gravatar.com/avatar/b890e13658e390e8dbf289304ea863589f6679161577e2bbb16a7626a267950b?d=mp&s=160"},"body":"Hello,\n\nMy high level problem is to speed up git commit on a large repository stored on NFS filesystem. I see via strace that it is slow because it makes a large number (~50,000) of lstat() calls in serial. Every call is a round-trip to the NFS server.\n\nI do not understand why git commit must call lstat() on every file in the repository, even when I specify the name of the file I want to commit on the command line. Can somebody explain why it must call lstat on every file?\n\nMy command-line looks like this: git commit -uno -o -m asdf file-to-commit.txt\n\nSecondly, are there any optimizations I can make to avoid this behavior?\n\nThanks,\nMatt\n\n"},{"id":"310666","messageId":"xmqq8tppo92x.fsf@gitster.mtv.corp.google.com","threadId":"45014","inReplyTo":"DA0A42D68346B1469147552440A645039A9C56D4@ORSMSX115.amr.corp.intel.com","subject":"Re: git commit results in many lstat()s","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-01T22:11:18Z","receivedAt":"2017-02-01T22:11:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Gumbel, Matthew K\" writes:\n\n> I do not understand why git commit must call lstat() on every file\n> in the repository, even when I specify the name of the file I want\n> to commit on the command line.\n\nAssuming the \"COPYING\" and \"README.md\" files are already tracked:\n\n $ >COPYING\n $ >README.md\n $ git commit COPYING\n\nwould open an editor, in which you would see list of files under\n\"Changes to be committed\", \"Changes not staged for commit\", etc.\nAmong the second class you would see README.md listed.\n\nTo figure out what paths are \"changed\", without having to open all\nfiles and compare their contents with what is recorded in the commit\nyou are building on top of, we do lstat(2) to see if the timestamp\n(and other information in the inode) of the files are the same since\nyou checked them out of HEAD.\n\n $ git commit --no-status COPYING\n\nwould reduce the number of lstat(2) somewhat, because the codepath\nis told that it does not have to make the list to be shown in the\neditor. So would\n\n $ git commit -m \"empty COPYING\" COPYING\n\nThese two only halve the number of lstat(2), by taking advantage of\nthe fact that the list of \"modified files\" does not have to be\nbuilt. There probably are other things that can be optimized.\n\n"},{"id":"310669","messageId":"DA0A42D68346B1469147552440A645039A9C57D6@ORSMSX115.amr.corp.intel.com","threadId":"45014","inReplyTo":"xmqq8tppo92x.fsf@gitster.mtv.corp.google.com","subject":"RE: git commit results in many lstat()s","fromName":"Gumbel, Matthew K","fromEmail":"matthew.k.gumbel@intel.com","sentAt":"2017-02-01T22:26:05Z","receivedAt":"2017-02-01T22:26:12Z","isPatch":false,"sender":{"key":"matthew.k.gumbel@intel.com","avatar":"https://gravatar.com/avatar/b890e13658e390e8dbf289304ea863589f6679161577e2bbb16a7626a267950b?d=mp&s=160"},"body":"\"Junio C Hamano\" writes:\n> There probably are other things that can be optimized.\n\nYes, I think that when the user passes --only flag to git-commit, then git does not\nneed to call refresh_cache() in prepare_index() in builtin/commit.c.\n\nI may experiment with that. Do you see any downside or negative side-effects?\n\nMatt\n"},{"id":"310688","messageId":"xmqq60ktmpwl.fsf@gitster.mtv.corp.google.com","threadId":"45014","inReplyTo":"DA0A42D68346B1469147552440A645039A9C57D6@ORSMSX115.amr.corp.intel.com","subject":"Re: git commit results in many lstat()s","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-02-01T23:50:50Z","receivedAt":"2017-02-01T23:50:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Gumbel, Matthew K\" writes:\n\n> \"Junio C Hamano\" writes:\n>> There probably are other things that can be optimized.\n>\n> Yes, I think that when the user passes --only flag to git-commit, then git does not\n> need to call refresh_cache() in prepare_index() in builtin/commit.c.\n>\n> I may experiment with that. Do you see any downside or negative side-effects?\n\nThere may be other fallouts, but one that immediately comes to mind\nis that it may break pre-commit hook.\n\nWhen we get \"--only\", we prepare an temporary index to create the\ncommit out of, and give it to the pre-commit hook. The hook expects\nthat the cached stat information is up-to-date, iow, it does not\nhave to do 'update-index --refresh' before using plumbing commands\nlike \"diff-index\" to do its own inspection of the working tree.\n"},{"id":"310690","messageId":"DA0A42D68346B1469147552440A645039A9C5929@ORSMSX115.amr.corp.intel.com","threadId":"45014","inReplyTo":"xmqq60ktmpwl.fsf@gitster.mtv.corp.google.com","subject":"RE: git commit results in many lstat()s","fromName":"Gumbel, Matthew K","fromEmail":"matthew.k.gumbel@intel.com","sentAt":"2017-02-02T00:14:30Z","receivedAt":"2017-02-02T00:14:47Z","isPatch":false,"sender":{"key":"matthew.k.gumbel@intel.com","avatar":"https://gravatar.com/avatar/b890e13658e390e8dbf289304ea863589f6679161577e2bbb16a7626a267950b?d=mp&s=160"},"body":"\"Junio C Hamano writes:\n\n\"Gumbel, Matthew K\" writes:\n\n>> Yes, I think that when the user passes --only flag to git-commit, then git does not\n>> need to call refresh_cache() in prepare_index() in builtin/commit.c.\n>>\n>> I may experiment with that. Do you see any downside or negative side-effects?\n\n> There may be other fallouts, but one that immediately comes to mind\n> is that it may break pre-commit hook.\n\nIf pre-commit hook exists, we can fall-back to original behavior and call\nrefresh_cache(). Many repos will not have pre-commit hook and can \nbenefit from the speedup.\n\nI'm testing such a change locally. Git test suite seems to be running for quite\na while. Do you know any way to run it in parallel or otherwise speed it\nup?\n\nThanks,\nMatt\n"},{"id":"310691","messageId":"20170202002556.mn7pfu4442btru5v@genre.crustytoothpaste.net","threadId":"45014","inReplyTo":"DA0A42D68346B1469147552440A645039A9C5929@ORSMSX115.amr.corp.intel.com","subject":"Re: git commit results in many lstat()s","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2017-02-02T00:25:57Z","receivedAt":"2017-02-02T00:26:10Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Thu, Feb 02, 2017 at 12:14:30AM +0000, Gumbel, Matthew K wrote:\n> I'm testing such a change locally. Git test suite seems to be running for quite\n> a while. Do you know any way to run it in parallel or otherwise speed it\n> up?\n\nI usually do something like the following:\n\n make -j3 all && (cd t && GIT_PROVE_OPTS=-j3 make prove)\n\nThis, of course, requires that you have Perl's prove installed, which\nhas been part of core Perl since 5.10.1.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"}]}