Re: Cygwin can't handle huge packfiles?

From: Christopher Faylor <me@cgf.cx>
Date: Apr 6, 2006, 00:34 UTC
Message-ID: <20060406003449.GA7174@trixie.casa.cgf.cx>
In-Reply-To: <20060405232739.GA18121@nospam.com>

On Thu, Apr 06, 2006 at 01:27:39AM +0200, Rutger Nijlunsing wrote:

Show 25 quoted lines

>On Wed, Apr 05, 2006 at 05:08:44PM -0400, Christopher Faylor wrote:
>> On Wed, Apr 05, 2006 at 04:14:20PM +0200, Johannes Schindelin wrote:
>> >> Inspired by a patch of Alex Riesen (thanks, Alex), I tried to use the
>> >> regular mmap for mapping pack files, only to discover that I compile
>> >> without defining "NO_MMAP", so I've been using the stock mmap all
>> >> along. So now I'm thinking that the cygwin mmap also does a
>> >> malloc-and-read, just like git does with NO_MMAP. So I'll continue to
>> >> investigate in that direction.
>> >
>> >I think cygwin's mmap() is based on the Win32 API equivalent, which could 
>> >mean that it *is* memory mapped, but in a special area (which is smaller 
>> >than 1.5 gigabyte). In this case, it would make sense to limit the pack 
>> >size, thereby having several packs, and mmap() them as they are needed.
>> 
>> Yes, cygwin's mmap uses CreateFileMapping and MapViewOfFile.  IIRC,
>> Windows might have a 2G limitation lurking under the hood somewhere but
>> I think that might be tweakable with some registry setting.
>
>Windows places its DLLs criss-cross through the memory space because
>every DLL on the system has its own preferred place to be loaded (the
>base address). This severely limits the amount of largest contiguous
>memory block available, which is needed for one mmap() I think.
>
>Several solutions exist:
>  - enlarge the address space with the /3GB boot flag in boot.ini

Thanks.  The 3GB boot flag is what I was trying to remember.

>  - rebase all DLLs with REBASE.EXE (part of platform sdk) .
>    Just make them the same and fix them to a low address.
>    Problem is rebasing system dlls since those are locked by the system.

Cygwin has its own version of rebase and a method for rebasing all of the dlls in the distribution. Using that may help squeeze out a little bit of memory.

>  - at start of program before other DLLs are loaded,
>    reserve an as large part of the memory as possible with
>    VirtualAlloc()

Cygwin actually uses this trick to try to push DLLs into their right locations after a fork. It sort of works but sometimes, in a child proccess, Windows puts "stuff" in locations previously occupied by a DLL. I could swear that it does that just to be annoying...

There is a chicken/egg problem here in that Cygwin uses Doug Lea's malloc and that version of malloc will use mmap when sbrk() fails -- as it is apt to do when allocating gigabytes of memory. So, using malloc is not a way to avoid mmap.

cgf