git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Excessive mmap [was Git server eats all memory]

From
Duy Nguyen <pclouds@gmail.com>
Date
Jun 20, 2018, 14:53 UTC
Message-ID
<CACsJy8AeFfi_k9vMm71BTTF-3GQMFPOTXMhfwqKWT-7RFVjoNw@mail.gmail.com>
In-Reply-To
<wesy6cgm6wd.fsf_-_@kanis.fr>

On Tue, Jun 19, 2018 at 10:27 PM Ivan Kanis <expire-by-2010-08-14@kanis.fr> wrote:

Show 20 quoted lines
>
> Dmitry Potapov <dpotapov@gmail.com> wrote:
>
> > On Fri, Aug 06, 2010 at 07:23:17PM +0200, Ivan Kanis wrote:
> >>
> >> I expected the malloc to take 4G but was surprised it didn't. It seems
> >> to be mmap taking all the memory. I am not familiar with that function,
> >> it looks like it's mapping memory to a file... Is it reasonable to mmap
> >> so much memory?
> >
> > AFAIK, Git does not need to mmap the whole pack to memory, but it
> > is more efficient to mmap the whole pack wherever possible, because
> > it has a completely random access, so if you store only one sliding
> > window, you will have to re-read it many times. Besides, mmap size
> > does not mean that so much physical memory is used. Pages should
> > be loaded when they are necessary, and if you have more than one
> > client cloning the same repo, this memory should be shared by them.
>
> I have clone identical repositories and the system starts to swap. I
> think it shows that cloning two repository doesn't share mmap.

I doubt it (assuming you're on linux). If you suspect this, configure core.packedGitWindowSize to reduce the mmap size. There are lots of other things in a cloning process that do not share (is this client or server btw?) and things could add up. -- Duy

Previous: Avery PennarunNext: Robin H. Johnson
Message 21 of 28 in “Git server eats all memory”
  1. Ivan KanisAug 4, 2010
  2. Matthieu MoyAug 4, 2010
  3. Ivan KanisAug 4, 2010
  4. Avery PennarunAug 4, 2010
  5. Ivan KanisAug 5, 2010
  6. Jared HanceAug 5, 2010
  7. Nguyen Thai Ngoc DuyAug 6, 2010
  8. Nguyen Thai Ngoc DuyAug 6, 2010
  9. Jakub NarebskiAug 6, 2010
  10. Ivan KanisAug 6, 2010
  11. Dmitry PotapovAug 7, 2010
  12. Excessive mmap [was Git server eats all memory]Ivan Kanis, Aug 9, 2010
  13. Dmitry PotapovAug 9, 2010
  14. Ivan KanisAug 9, 2010
  15. Avery PennarunAug 9, 2010
  16. Tomas CarneckyAug 9, 2010
  17. Avery PennarunAug 9, 2010
  18. Dmitry PotapovAug 9, 2010
  19. Ivan KanisAug 11, 2010
  20. Avery PennarunAug 11, 2010
  21. Duy NguyenJun 20, 2018
  22. Robin H. JohnsonAug 10, 2010
  23. Sverre RabbelierAug 10, 2010
  24. Sam VilainAug 11, 2010
  25. Ivan KanisAug 11, 2010
  26. Nguyen Thai Ngoc DuyAug 9, 2010
  27. Ivan KanisAug 9, 2010
  28. Dmitry PotapovAug 17, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.