Re: Cygwin can't handle huge packfiles?

From: Linus Torvalds <torvalds@osdl.org>
Date: Apr 3, 2006, 14:33 UTC
Message-ID: <Pine.LNX.4.64.0604030730040.3781@g5.osdl.org>
In-Reply-To: <Pine.LNX.4.63.0604031521170.4011@wbgn013.biozentrum.uni-wuerzburg.de>

On Mon, 3 Apr 2006, Johannes Schindelin wrote:

Show 5 quoted lines

> 
> The problem is not mmap() on cygwin, but that a fork() has to jump through 
> loops to reinstall the open file descriptors on cygwin. If the 
> corresponding file was deleted, that fails. Therefore, we work around that 
> on cygwin by actually reading the file into memory, *not* mmap()ing it.

Well, we could actually do a _real_ mmap on pack-files. The pack-files are much better mmap'ed - there we don't _want_ them to be removed while we're using them. It was the index file etc that was problematic.

Maybe the cygwin fake mmap should be triggered only for the index (and possibly the individual objects - if only because there doing a malloc+read may actually be faster).

Using malloc+read on pack-files is pretty wasteful, since we usually only use a very small part of them (ie if we have a 1.5GB pack-file, it's sad to read all of it, when we'd usually actually access just a small small fraction of it).

That said, I think git _does_ have problems with large pack-files. We have some 32-bit issues etc, and just virtual address space things. So for now, it's probably best to limit pack-files to the few-hundred-meg size, and create serveral smaller ones rather than one huge one.

		Linus