git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] RFC: git lazy clone proof-of-concept

From
Jon Smirl <jonsmirl@gmail.com>
Date
Feb 10, 2008, 17:01 UTC
Message-ID
<9e4733910802100901m729b0cdfg85ccc0ca77011249@mail.gmail.com>
In-Reply-To
<alpine.LSU.1.00.0802101640570.11591@racer.site>
On 2/10/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
Show 40 quoted lines
> Hi,
>
> On Sat, 9 Feb 2008, Nicolas Pitre wrote:
>
> > On Sat, 9 Feb 2008, Jan Holesovsky wrote:
> >
> > > On Friday 08 February 2008 20:00, Jakub Narebski wrote:
> > >
> > > > Both Mozilla import, and GCC import were packed below 0.5 GB. Warning:
> > > > you would need machine with large amount of memory to repack it
> > > > tightly in sensible time!
> > >
> > > As I answered elsewhere, unfortunately it goes out of memory even on 8G
> > > machine (x86-64), so...  But still trying.
> >
> > Try setting the following config variables as follows:
> >
> >       git config pack.deltaCacheLimit 1
> >       git config pack.deltaCacheSize 1
> >       git config pack.windowMemory 1g
> >
> > That should help keeping memory usage somewhat bounded.
>
> I tried that:
>
> $ git config pack.deltaCacheLimit 1
> $ git config pack.deltaCacheSize 1
> $ git config pack.windowMemory 2g
> $ #/usr/bin/time git repack -a -d -f --window=250 --depth=250
> $ du -s objects/
> 2548137 objects/
> $ /usr/bin/time git repack -a -d -f --window=250 --depth=250
> Counting objects: 2477715, done.
> fatal: Out of memory, malloc failed411764)
> Command exited with non-zero status 1
> 9356.95user 53.33system 2:38:58elapsed 98%CPU (0avgtext+0avgdata
> 0maxresident)k
> 0inputs+0outputs (31929major+18088744minor)pagefaults 0swaps
>
> Note that this is on a 2.4GHz Quadcode CPU with 3.5GB RAM.

Turning on multi-core support greatly increases the memory consumption; at least double the single thread case.

Going over the original repository and deleting (get all copies out of the history) those giant i18n files generated by programs than Sean refers to would be my first step. If you have 5,000 revisions of a 10MB file I suspect it would take a huge amount of memory to pack. Plus you have to copy all of that pointless history around.

Show 12 quoted lines
>
> I'm retrying with smaller values, but at over 2.5 hours per try, this is
> getting tedious.
>
> Ciao,
> Dscho
>
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
-- 
Jon Smirl
jonsmirl@gmail.com
Previous: Johannes SchindelinNext: Johannes Schindelin
Message 38 of 85 in “RFC: git lazy clone proof-of-concept”
  1. RFC: git lazy clone proof-of-conceptJan Holesovsky, Feb 8, 2008
  2. Nicolas PitreFeb 8, 2008
  3. Jan HolesovskyFeb 9, 2008
  4. Mike HommeyFeb 9, 2008
  5. Nicolas PitreFeb 9, 2008
  6. Marco CostalbaFeb 10, 2008
  7. Johannes SchindelinFeb 10, 2008
  8. David SymondsFeb 10, 2008
  9. Johannes SchindelinFeb 10, 2008
  10. Nicolas PitreFeb 10, 2008
  11. Johannes SchindelinFeb 10, 2008
  12. Harvey HarrisonFeb 8, 2008
  13. Jan HolesovskyFeb 9, 2008
  14. Johannes SchindelinFeb 8, 2008
  15. Mike HommeyFeb 8, 2008
  16. Johannes SchindelinFeb 8, 2008
  17. Jan HolesovskyFeb 9, 2008
  18. Jakub NarebskiFeb 8, 2008
  19. Jon SmirlFeb 8, 2008
  20. Nicolas PitreFeb 8, 2008
  21. Andreas EricssonFeb 11, 2008
  22. 1/2 pack-objects: Allow setting the #threads equal to #cpus automaticallyBrandon Casey, Feb 12, 2008
  23. Andreas EricssonFeb 12, 2008
  24. Harvey HarrisonFeb 8, 2008
  25. Jon SmirlFeb 8, 2008
  26. Harvey HarrisonFeb 8, 2008
  27. Jon SmirlFeb 8, 2008
  28. Jan HolesovskyFeb 9, 2008
  29. Nicolas PitreFeb 10, 2008
  30. SeanFeb 10, 2008
  31. Nicolas PitreFeb 10, 2008
  32. SeanFeb 10, 2008
  33. Jakub NarebskiFeb 11, 2008
  34. Nicolas PitreFeb 11, 2008
  35. Jakub NarebskiFeb 11, 2008
  36. Joachim B HagaFeb 10, 2008
  37. Johannes SchindelinFeb 10, 2008
  38. Jon SmirlFeb 10, 2008
  39. Johannes SchindelinFeb 10, 2008
  40. Johannes SchindelinFeb 10, 2008
  41. Nicolas PitreFeb 10, 2008
  42. Jon SmirlFeb 10, 2008
  43. Johannes SchindelinFeb 12, 2008
  44. Nicolas PitreFeb 12, 2008
  45. Linus TorvaldsFeb 12, 2008
  46. Jon SmirlFeb 12, 2008
  47. Linus TorvaldsFeb 12, 2008
  48. Linus TorvaldsFeb 12, 2008
  49. Jon SmirlFeb 12, 2008
  50. Linus TorvaldsFeb 12, 2008
  51. Jon SmirlFeb 12, 2008
  52. Johannes SchindelinFeb 14, 2008
  53. Jakub NarebskiFeb 14, 2008
  54. Nicolas PitreFeb 14, 2008
  55. Johannes SchindelinFeb 14, 2008
  56. Jakub NarebskiFeb 14, 2008
  57. Johannes SchindelinFeb 14, 2008
  58. Brian DowningFeb 14, 2008
  59. Brian DowningFeb 14, 2008
  60. Johannes SchindelinFeb 15, 2008
  61. Nicolas PitreFeb 15, 2008
  62. Shawn O. PearceFeb 17, 2008
  63. Junio C HamanoFeb 17, 2008
  64. Nicolas PitreFeb 17, 2008
  65. Jakub NarebskiFeb 15, 2008
  66. Jan HolesovskyFeb 15, 2008
  67. Brandon CaseyFeb 14, 2008
  68. Jan HolesovskyFeb 15, 2008
  69. Nicolas PitreFeb 10, 2008
  70. Brandon CaseyFeb 14, 2008
  71. Johannes SchindelinFeb 14, 2008
  72. Nicolas PitreFeb 14, 2008
  73. Jakub NarebskiFeb 11, 2008
  74. Johannes SchindelinFeb 8, 2008
  75. Jakub NarebskiFeb 8, 2008
  76. Johannes SchindelinFeb 8, 2008
  77. Mike HommeyFeb 8, 2008
  78. Johannes SchindelinFeb 8, 2008
  79. Mike HommeyFeb 8, 2008
  80. Johannes SchindelinFeb 8, 2008
  81. Mike HommeyFeb 8, 2008
  82. Jan HudecFeb 9, 2008
  83. Jan HolesovskyFeb 9, 2008
  84. 2/2 pack-objects: Default to zero threads, meaning auto-assign to #cpusBrandon Casey, Feb 12, 2008
  85. Nicolas PitreFeb 12, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.