Re: Cygwin can't handle huge packfiles?

From: Rutger Nijlunsing <rutger@nospam.com>
Date: Apr 5, 2006, 23:27 UTC
Message-ID: <20060405232739.GA18121@nospam.com>
In-Reply-To: <20060405210844.GN26780@trixie.casa.cgf.cx>

On Wed, Apr 05, 2006 at 05:08:44PM -0400, Christopher Faylor wrote:

Show 16 quoted lines

> On Wed, Apr 05, 2006 at 04:14:20PM +0200, Johannes Schindelin wrote:
> >> Inspired by a patch of Alex Riesen (thanks, Alex), I tried to use the
> >> regular mmap for mapping pack files, only to discover that I compile
> >> without defining "NO_MMAP", so I've been using the stock mmap all
> >> along. So now I'm thinking that the cygwin mmap also does a
> >> malloc-and-read, just like git does with NO_MMAP. So I'll continue to
> >> investigate in that direction.
> >
> >I think cygwin's mmap() is based on the Win32 API equivalent, which could 
> >mean that it *is* memory mapped, but in a special area (which is smaller 
> >than 1.5 gigabyte). In this case, it would make sense to limit the pack 
> >size, thereby having several packs, and mmap() them as they are needed.
> 
> Yes, cygwin's mmap uses CreateFileMapping and MapViewOfFile.  IIRC,
> Windows might have a 2G limitation lurking under the hood somewhere but
> I think that might be tweakable with some registry setting.

Windows places its DLLs criss-cross through the memory space because every DLL on the system has its own preferred place to be loaded (the base address). This severely limits the amount of largest contiguous memory block available, which is needed for one mmap() I think.

Several solutions exist:
  - enlarge the address space with the /3GB boot flag in boot.ini
  - rebase all DLLs with REBASE.EXE (part of platform sdk) .
    Just make them the same and fix them to a low address.
    Problem is rebasing system dlls since those are locked by the system.
  - at start of program before other DLLs are loaded,
    reserve an as large part of the memory as possible with
    VirtualAlloc()

-- 
Rutger Nijlunsing ---------------------------------- eludias ed dse.nl
never attribute to a conspiracy which can be explained by incompetence
----------------------------------------------------------------------