Re: large files and low memory

From: Shawn Pearce <spearce@spearce.org>
Date: Oct 4, 2010, 18:05 UTC
Message-ID: <AANLkTimbdrAqoWMxiteT5zNYmwHp8M698BEv1FLuiAxx@mail.gmail.com>
In-Reply-To: <20101004092046.GA4382@nibiru.local>

On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt <weigelt@metux.de> wrote:

Show 5 quoted lines

>
> when adding files which are larger than available physical memory,
> git performs very slow. Perhaps it has to do with git's mmap()ing
> the whole file. Is there any way to do it w/o mmap (hoping that
> might perform a bit better) ?

The mmap() isn't the problem. Its the allocation of a buffer that is larger than the file in order to hold the result of deflating the file before it gets written to disk. When the file is bigger than physical memory, the kernel has to page in parts of the file as well as swap in and out parts of that allocated buffer to hold the deflated file.

This is a known area in Git where big files aren't handled well.

-- 
Shawn.