git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Git performance results on a large repository

From
Joey Hess <joey@kitenet.net>
Date
Feb 4, 2012, 05:07 UTC
Message-ID
<20120204050712.GA2460@gnu.kitenet.net>
In-Reply-To
<CB5074CF.3AD7A%joshua.redstone@fb.com>
Joshua Redstone wrote:
> The test repo has 4 million commits, linear history and about 1.3 million
> files.

Have you tried separating these two factors, to see how badly each is affecting performance?

If the number of commits is the problem (seems likely for git blame at least), a shallow clone would avoid that overhead.

I think that git often writes .git/index inneficiently when staging files (though your `git add` is pretty fast) and committing. It rewrites the whole file to .git/index.lck and the renames it over .git/index at the end. I have code that keeps a journal of changes to avoid rewriting the index repeatedly, but it's application specific. Fixing git to write the index more intelligently is something I'd like to see.

Hint for git status: `git status .` in a smaller subdirectory will be much faster than the default that stats everything.

-- 
see shy jo
Previous: Zeki MokhtarzadaNext: Nguyen Thai Ngoc Duy
Message 11 of 34 in “Git performance results on a large repository”
  1. Joshua RedstoneFeb 3, 2012
  2. Ævar Arnfjörð BjarmasonFeb 3, 2012
  3. Joshua RedstoneFeb 3, 2012
  4. Sam VilainFeb 3, 2012
  5. Sam VilainFeb 3, 2012
  6. Nguyen Thai Ngoc DuyFeb 7, 2012
  7. Matt GrahamFeb 3, 2012
  8. Evgeny SazhinFeb 4, 2012
  9. Chris LeeFeb 3, 2012
  10. Zeki MokhtarzadaFeb 4, 2012
  11. Joey HessFeb 4, 2012
  12. Nguyen Thai Ngoc DuyFeb 4, 2012
  13. Joshua RedstoneFeb 4, 2012
  14. Nguyen Thai Ngoc DuyFeb 5, 2012
  15. Joey HessFeb 6, 2012
  16. Nguyen Thai Ngoc DuyFeb 7, 2012
  17. Joshua RedstoneFeb 9, 2012
  18. Nguyen Thai Ngoc DuyFeb 10, 2012
  19. Christian CouderFeb 10, 2012
  20. Nguyen Thai Ngoc DuyFeb 10, 2012
  21. David MohsFeb 6, 2012
  22. Matt GrahamFeb 6, 2012
  23. Joshua RedstoneFeb 6, 2012
  24. Greg TroxelFeb 6, 2012
  25. david@lang.hmFeb 7, 2012
  26. Sam VilainFeb 6, 2012
  27. Joshua RedstoneFeb 4, 2012
  28. Tomas CarneckyFeb 5, 2012
  29. Nguyen Thai Ngoc DuyFeb 5, 2012
  30. slinkyFeb 4, 2012
  31. Greg TroxelFeb 4, 2012
  32. david@lang.hmFeb 5, 2012
  33. David BarrFeb 5, 2012
  34. Emanuele ZattinFeb 7, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.