Re: win2k/cygwin cannot handle even moderately sized packs

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Nov 13, 2006, 12:45 UTC
Message-ID: <Pine.LNX.4.63.0611131333000.13772@wbgn013.biozentrum.uni-wuerzburg.de>
In-Reply-To: <81b0412b0611070302h50541cd5mf0758afe0d6befda@mail.gmail.com>

Hi,

On Tue, 7 Nov 2006, Alex Riesen wrote:

Show 12 quoted lines

> For me, it fails even on 332Mb pack:
> 
> $ git reset --hard 61bb7fcb
> fatal: packfile .git/objects/pack/pack-ad37...pack cannot be mapped.
> 
> Instrumenting the code reveals that it fails on 348876870 bytes.
> Strangely enough, a cygwin program which just reads that pack
> many times without freeing the mem goes up to 1395507480 (1330Mb).
> 
> If I replace the malloc (cygwin) with native VirtualAlloc(MEM_COMMIT)
> it reports that "Not enough storage is available to process this command",
> which is just ENOMEM, I think.

This looks to me as if you have NO_MMAP=1 set in your Makefile (which I also do automatically when compiling on cygwin).

The old problem: Windows does not have fork.

<digression> And before somebody starts cygwin bashing: don't. It is not cygwin's problem, it is Windows'. The cygwin people worked long and hard on something truly useful, and it helps me _every_ time I have to work on a Windows platform (which _is_ utter crap). Some problems of Windows are so unhideable though, that even cygwin cannot work around them. </digression>

Cygwin provides a mmap(), which works remarkably well even with the emulated fork(), but one thing is not possible: since mmap()ed files have to be _reopened_ after a fork(), and git uses the open-temp-file-then-delete-it-but-continue-to-use-it paradigm quite often, we work around it by setting NO_MMAP=1. Again, this is _not_ Cygwin's fault!

And I think that a mmap() of 332MB would not fail.

Long time ago (to be precise, July 18th), Linus suggested (in Message-Id: <Pine.LNX.4.64.0607180837260.3386@evo.osdl.org>) to find out which mmap() calls are _not_ used before a fork(), and not emulate them by malloc().

I never came around to do that, but maybe others do?

Ciao, Dscho