Re: large files and low memory

From: Enrico Weigelt <weigelt@metux.de>
Date: Oct 5, 2010, 22:51 UTC
Message-ID: <20101005225150.GA27633@nibiru.local>
In-Reply-To: <alpine.LFD.2.00.1010051504530.3107@xanadu.home>

* Nicolas Pitre <nico@fluxnic.net> wrote:

Show 10 quoted lines

> > > The mmap() isn't the problem.  Its the allocation of a buffer that is
> > > larger than the file in order to hold the result of deflating the file
> > > before it gets written to disk.  
> > > When the file is bigger than physical memory, the kernel has to
> > > page in parts of the file as well as swap in and out parts of
> > > that allocated buffer to hold the deflated file.
> > 
> > What are the access pattern of these memory areas ?
> 
> Perfectly linear.

In this case, I wonder why my machine goes into thrashing so easily (P3 w/ 256MB ram). Seems the mmu/paging code doesnt recognize that the previously-used pages can be kicked-off quickly ;-o Perhaps I should talk to the kernel folks.

> > Perhaps madvise() could help ?
> 
> Perhaps.

hmm, so we should try it ;-p where'd be the right place to add it ?

cu

-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------