git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Git performance results on a large repository

From
David Barr <davidbarr@google.com>
Date
Feb 5, 2012, 11:24 UTC
Message-ID
<CAFfmPPMcS5Y8hoXyR8LY637z_f-A1XgrH6m+YUsCv8gmGjtr3w@mail.gmail.com>
In-Reply-To
<alpine.DEB.2.02.1202042026280.6541@asgard.lang.hm>
On Sun, Feb 5, 2012 at 3:30 PM,  <david@lang.hm> wrote:
Show 20 quoted lines
> On Fri, 3 Feb 2012, Joshua Redstone wrote:
>
>> The test repo has 4 million commits, linear history and about 1.3 million
>> files.  The size of the .git directory is about 15GB, and has been
>> repacked with 'git repack -a -d -f --max-pack-size=10g --depth=100
>> --window=250'.  This repack took about 2 days on a beefy machine (I.e.,
>> lots of ram and flash).  The size of the index file is 191 MB.
>
>
> This may be a silly thought, but what if instead of one pack file of your
> entire history (4 million commits) you create multiple packs (say every half
> million commits) and mark all but the most recent pack as .keep (so that
> they won't be modified by a repack)
>
> that way things that only need to worry about recent history (blame, etc)
> will probably never have to go past the most recent pack file or two
>
> I may be wrong, but I think that when git is looking for 'similar files' for
> delta compression, it limits it's search to the current pack, so this will
> also keep you from searching the entire project history.

I don't know if there is an easy way to determine with the with the current tools in git but one useful statistic for tuning packing performance is the size of the largest component in the delta-chain graph. The significance of this number is that the product of window-size and maximum depth need not be larger than it. I've found that with some older repositories I could have a depth as low as 3 and still get good performance from a moderate window size.

-- David Barr

Previous: david@lang.hmNext: Emanuele Zattin
Message 33 of 34 in “Git performance results on a large repository”
  1. Joshua RedstoneFeb 3, 2012
  2. Ævar Arnfjörð BjarmasonFeb 3, 2012
  3. Joshua RedstoneFeb 3, 2012
  4. Sam VilainFeb 3, 2012
  5. Sam VilainFeb 3, 2012
  6. Nguyen Thai Ngoc DuyFeb 7, 2012
  7. Matt GrahamFeb 3, 2012
  8. Evgeny SazhinFeb 4, 2012
  9. Chris LeeFeb 3, 2012
  10. Zeki MokhtarzadaFeb 4, 2012
  11. Joey HessFeb 4, 2012
  12. Nguyen Thai Ngoc DuyFeb 4, 2012
  13. Joshua RedstoneFeb 4, 2012
  14. Nguyen Thai Ngoc DuyFeb 5, 2012
  15. Joey HessFeb 6, 2012
  16. Nguyen Thai Ngoc DuyFeb 7, 2012
  17. Joshua RedstoneFeb 9, 2012
  18. Nguyen Thai Ngoc DuyFeb 10, 2012
  19. Christian CouderFeb 10, 2012
  20. Nguyen Thai Ngoc DuyFeb 10, 2012
  21. David MohsFeb 6, 2012
  22. Matt GrahamFeb 6, 2012
  23. Joshua RedstoneFeb 6, 2012
  24. Greg TroxelFeb 6, 2012
  25. david@lang.hmFeb 7, 2012
  26. Sam VilainFeb 6, 2012
  27. Joshua RedstoneFeb 4, 2012
  28. Tomas CarneckyFeb 5, 2012
  29. Nguyen Thai Ngoc DuyFeb 5, 2012
  30. slinkyFeb 4, 2012
  31. Greg TroxelFeb 4, 2012
  32. david@lang.hmFeb 5, 2012
  33. David BarrFeb 5, 2012
  34. Emanuele ZattinFeb 7, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.