Re: pack operation is thrashing my server
- From
Shawn O. Pearce <spearce@spearce.org>
- Date
- Aug 13, 2008, 02:57 UTC
- Message-ID
- <20080813025749.GA5855@spearce.org>
- In-Reply-To
- <20080813025027.GG1366@one.firstfloor.org>
Andi Kleen <andi@firstfloor.org> wrote:
Show 11 quoted lines
> > If you have only 256 MB of RAM, I'm afraid the machine dives into swap > > the moment it attempts to process that single 150-MB object during > > repacking. Objects are always allocated entirely, including the > > deflated and inflated copy at some point. Making git handle partial > > objects in memory would add complexity all over the map so I don't think > > it'll ever be implemented nor be desirable. > > If the access pattern is sequential and not much reuse it might be possible > to madvise() strategically to do prefetch and early unmap of not used > anymore data. I used that successfully in a few programs in the past that did > aggressive mmap on very large files.
We actually do something better where we can. However parts of Git assume that it can get back a contiguous block of memory which contains the entire file content, decompressed. The data is stored on disk compressed, so we cannot just mmap the data from disk.
-- Shawn.