From: Joshua Jensen <jjensen@workspacewhiz.com>
Date: Mon, 04 Oct 2010 18:24:43 GMT
Subject: Re: large files and low memory
Message-ID: <4CAA1BEB.3050908@workspacewhiz.com>
In-Reply-To: <AANLkTimbdrAqoWMxiteT5zNYmwHp8M698BEv1FLuiAxx@mail.gmail.com>

  ----- Original Message -----
From: Shawn Pearce
Date: 10/4/2010 12:05 PM
> On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt<weigelt@metux.de>  wrote:
>> when adding files which are larger than available physical memory,
>> git performs very slow. Perhaps it has to do with git's mmap()ing
>> the whole file. Is there any way to do it w/o mmap (hoping that
>> might perform a bit better) ?
> The mmap() isn't the problem.  Its the allocation of a buffer that is
> larger than the file in order to hold the result of deflating the file
> before it gets written to disk.  When the file is bigger than physical
> memory, the kernel has to page in parts of the file as well as swap in
> and out parts of that allocated buffer to hold the deflated file.
>
> This is a known area in Git where big files aren't handled well.
As a curiosity, I've always done streaming decompression with zlib using 
minimal buffer sizes (64k, perhaps).  I'm sure there is good reason why 
Git doesn't do this (delta application?).  Do you know what it is?

Josh